Growth Snack: Write the Report Before You Run the Experiment

Episode 61 February 15, 2022 00:08:30
Growth Snack: Write the Report Before You Run the Experiment
The Breakout Growth Podcast
Growth Snack: Write the Report Before You Run the Experiment

Feb 15 2022 | 00:08:30


Show Notes

How can growth teams win by borrowing a bit from their engineering counterparts?


Find out in this week’s Growth Snack: The Breakout Growth Podcast Short. Here, Sean Ellis and Ethan Garr discuss the idea that growth teams win when they “Write the Report Before they Run the Experiment”.  Ethan says growth teams that do this run better experiments, more frequently, with more predictable outcomes.


Sean likes the idea because he has seen just how difficult it can be to get high-tempo testing to take flight. Because this technique forces teams to start with the end in mind as they run experiments, he believes it is a good habit to get into if you are looking to improve your growth trajectory.


If you want to learn more about applying this concept after you listen, visit, “Write The Report Before You Run The Experiment” on

We discussed: 

* Writing the report first is a good idea if you want to test more effectively (28.81)

* Why faster testing shouldn’t come at the expense of quality learnings (01:18)

* Documenting what you are going to do; a lesson from GoPractice!  (03:48)

*  Begin with the end in mind to think ahead and drive test/learn cycles (04:26)

*  Who should build this habit (05:34)

View Full Transcript

Episode Transcript

Speaker 0 00:00:01 If you are leading growth, building a startup or looking to ladder up your skills, then you're probably really busy. So every other week tune in to growth snack, the breakout growth podcast, short where Ethan Gar and I share one key learning to help you on your journey to break out growth, success. It's food for thought for anyone hungry for growth. Alrighty. Then let's talk about the concept of write the report before you run the experiment. What is it and why is it important? Speaker 1 00:00:28 It's a really simple concept. My hypothesis is that if you're leading growth and you want to run better experiments more frequently with more predictable outcomes, then you should work to build a habit around writing the report before you run the experiment. And why is that important? Well, I think we're all driven to run high velocity testing. We want to run more and more tests so we can out test our competitors and get those learnings. But it's only valuable if it actually produces quality learnings that are going to power, that flywheel where you take a learning and you apply it to what you're going to do next. Um, so if you write that report before you run the experiment, I think you're going to improve the odds that you're going to get those actionable insights that are going to matter and help them grow. Speaker 0 00:01:08 Right. Okay. So what happens if you don't do this? Speaker 1 00:01:10 Yeah. The danger is that instead of your experiments, turning guesses into facts, you're going to turn guesses into either more guesses or more unanswered questions and actually think it can be really a culture killing the state. If you do that. Um, while we, again, we're all trying to run these high velocity experiments. When your growth meetings turn into these unanswered questions where you're, everyone's pointing at each other saying, did we learn anything? Do we do, did we get the data? Does anyone understand what we learned? It really, it sets you back and you might think you're achieving something. Cause you're running a lot of tests, but they're not really experiments if you don't get anything out of them. Speaker 0 00:01:46 Right? Yeah. I've, I've definitely seen teams, particularly teams that go from not doing any testing. They suddenly start testing to follow my advice, to hold yourself accountable on how much testing you're doing. I really pushed high tempo testing for a long time. But what happens is that they start to just go through the motions of testing. They're like, okay, this is what we're supposed to do and almost become robotic around it. And uh, I think when you do that, you're just, you're not learning what you should be trying to learn from that testing. Speaker 1 00:02:17 I mean, I definitely blame you. This is definitely all your fault, but no, I mean, uh, truthfully, you know, you got me super excited about this idea of high tempo testing. I heard it from you, uh, you know, early on, I think maybe while you were still writing your book with Morgan brown and right away, I took it. I went back to my, my team and I was like, we're going to do this. And it got everyone excited, but we got ahead of our skis and I take full credit and blame for it. Um, we, uh, we kept making these unforced errors, you know, because we would run what we were calling an experiment, but we didn't have a fleshed out hypothesis or we didn't have the data. Or even if we had the data, we didn't have the skillset to actually understand that data. Speaker 1 00:02:56 And it really just, it was, you know, I think we're in of really setting ourselves back instead of moving ourselves forward. And I remember I was chatting with our director of engineering at the time and he was telling me about test-driven development, which is this idea. And I honestly I've stole a lot of this from, from this idea of test driven development, where you write your test case before you write your code and you do that because it has these forcing functions. It forces you to make sure that you're just writing code that meets the requirements instead of adding stuff on top of it, that's irrelevant or writing stuff. That's just irrelevant because you got excited about something else. Um, and it also has this neat forcing function of getting teams, talking to each other people, you know, talking, why are we doing this? Why does it matter? And you know, so I just sort of stole that and turn this into this thing. Thought on report driven testing where you write the, the report before you write the experiment. And I, as I piloted it with teams, I've really seen it make an impact. Speaker 0 00:03:50 Yeah. You know, this is really consistent with, uh, something we teach in the go practice simulator, which is when you're creating an experiment and you're building an experiment document. One of the key components of that document is what are we going to do based on the results of this experiment and it's, and it's often one of the most overlooked, uh, areas of an experiment document. And it's something I really learned from Oleg and, and have put into practice and seeing the value of that. But it, it actually reminds me of, uh, one of my favorite books over the years, uh, seven habits of highly effective people from Stephen Covey, where he talks about begin with the end in mind. And so I think if you're really intentional about what you're going to do, based on the learning from a test, you're much more likely to run that test in a way that you're going to get that learning. And the learning really comes from the report. So this fits really nicely with that concept of, of, you know, what, what is the outcome that we're shooting for, what we're going to, what are we going to do based on that outcome? And let's make sure that we know exactly what we're going to use in terms of information to make those decisions. Yeah, Speaker 1 00:05:02 I think Ali is really good about sort of running that sort of, you know, helping you understand how a systematic approach to this can really drive that end goal more efficiently and more effectively. I just, one of the things I really loved about go practice, but I, you know, I think with that too, is when you, when you start with that end in mind and you know where you're going, it just makes the whole process easier. I mean, if you think about it, like you and I are very anti-hero bureaucracy, we're not, we don't, I don't want people to write extra reports and extra documentation. You're going to have to do this anyway. But if you read this earlier on the process, it improves the odds that you're going to succeed. And look, I think, you know, you pointed out the spec and I mentioned, you know, early on for me with, uh, when I was leading a growth team, it was really important at the beginning. Speaker 1 00:05:47 And I think for most teams, this is going to be just good growth hygiene. It'll help you, um, just run more effective experimentation. But obviously if you have a really highly tuned experimentation process, I mean, if you're carrying Flanagan at HubSpot or Morgan, uh, brown, your co co-author of hacking growth at Shopify, you know, we've talked to them, they have incredible tools available to them to do just awesome experimentation. If they don't do this ahead of time, they might be okay because they have those tools. They have all the infrastructure to support their experiments. But especially if you're getting going in this, I, you know, I think this is just one of those simple things you can do to really improve your odds of success. Speaker 0 00:06:29 Yeah. So this is, this is really a quick introduction to this topic. And, and, uh, as we, as we talk about with growth snack, we're really trying to have this be food for thought that that gets the wheels turning so that you, you start to look for ways to improve how you and your company are approaching growth. But if somebody wants to go deeper and really dig deeper into this topic and, and think about specifically, how, how do I apply this? Where can they go for that information? Speaker 1 00:06:57 Yeah. I mean, I've put a lot of thought around this and I actually wrote a short article on our shared website, breakout Uh, it's called write the, write the report before you run the experiment. So easy to find if you go to our blog. Um, but yeah, just in general, I just want to let our audience know that, you know, we created breakout for them. It's a place where you and I share any thoughts we have that we think will help teams accelerate their growth trajectory. So, um, certainly, uh, check it out there. There's a lot more, it's a short article, but there's a lot more depth to this. And if you have any questions don't hesitate to reach Speaker 0 00:07:28 That's me. Absolutely. So that's our first, uh, gross snack. Um, this is something that, uh, is, is something we're hoping to do on a regular basis as part of the breakout growth, uh, podcast, but just a short version of it. Um, I consider it a pilot. So, um, we really want people's feedback on that. And, uh, if, if it's positive, that's great. We'll keep doing it. If, if you don't find it valuable, um, I think that's important as well. Um, so you can reach out to us through Twitter or LinkedIn, send us a message, or even through the breakout site, there's a link on there to contact us. Um, and, uh, yeah, if the feedback's good, then we'll, we'll start doing this every other week in between our, our longer episodes. And, uh, and you know, even as you, as you reach out, there might be some specific areas that you'd like us to cover. So let us know that in the messages as well, but thank you everybody for tuning in. Yeah. Speaker 1 00:08:26 Let's talk about gross. Made me hungry. Let's let's set it.

Other Episodes

Episode 40

January 19, 2021 00:56:46
Episode Cover

Unlocking Sustainable Growth: Expert Shares How to Dial In Product/Market Fit

Is it just luck that determines which companies successfully find and harness Product/Market Fit? Or are there specific strategies teams can employ to systematically...


Episode 43

March 05, 2021 01:00:26
Episode Cover

Is VR the Next Great Growth Frontier? FitXR’s Fitness App Makes the Case

Slip on your Oculus headset and FitXR will get your heart rate up. They are combining gaming, fitness, and virtual reality together to create...


Episode 20

March 19, 2020 00:51:05
Episode Cover

California’s Largest Cannabis Distributor Nabis CEO Shares His Growth Journey to Breakout Success

In this episode, Sean Ellis interviews Nabis Founder and CEO Vincent Ning to understand what is driving their breakout growth. Vincent tells the story...