Episode Transcript
Announcer 00:00:08 Welcome to the Breakout Growth Podcast, where Sean Ellis and Ethan Garr interview leaders from the world's fastest-growing companies to get to the heart of what's really driving their growth. And now here are your hosts, Sean Ellis and Ethan Garr.
Sean Ellis 00:00:25 All right. In this week's episode of the Breakout Growth Podcast, Ethan Garr and I chat with Jonas Klink, the Vice President of Product Management and Design at Thrive Market. So, Thrive is a subscription based online grocery delivery service on a mission to make healthy living easy and affordable for everyone. And Jonas is all about furthering that mission with the rigorous test/learn approach to growth that we all know and love here on the podcast. So what stood out for you, Ethan <laugh>?
Ethan Garr 00:00:55 Yeah, I, I think it is that discipline and rigor that Jonas is building and his team, it's just super admirable. Um, he brings this depth of experience from roles that companies like Google, eBay, Jet, and Walmart. Um, and it just seems to have informed this really smart, organized, just well-thought-out approach. And I think at the heart of that approach is building test velocity, which you and I are really passionate about. And I just love that formula. He shared Sean: velocity times win rate times average impact equals growth.
Sean Ellis 00:01:24 Yeah. I can't believe that I hadn't heard that expressed as that particular formula <laugh>. I asked him, he was the one who came up with that formula and he said, no, he's, he's heard others talk about it, but he should, he
Ethan Garr 00:01:34 Should have taken, he should have taken credit for
Sean Ellis 00:01:36 It. <laugh>. Well, I'm glad he didn't. So I'm gonna, I'm gonna repeat that. So velocity times win rate times average impact equals growth. And it's really hard to argue with that one. It just shows you that the more you optimize your testing approach, the more effective you can become. And I love that he talks about velocity in terms of learning. So it's learning velocity testing is important, but if you're not learning, it doesn't matter. So to me, that's where teams really need to build their momentum and find their stride. And he says that there is no failing in AB testing, just learning. So I love that as well. Um, you know, you need to embrace that if you're a growth leader, that that should be your mantra that just evangelize that there is no failing in AB testing, just learning and, uh, testing takes risk. It's just, it's natural. It takes risk. And, uh, with, uh, with risk is, um, you're not gonna be taking risk if you don't have that psychological safety that that's so critical to taking a risk.
Ethan Garr 00:02:37 Yeah, it's so true. And you know, I think that touches on one of the sort of bigger themes that we explored in the conversation, in the conversation, which is language really matters in leading teams. You know, there's some great dialogue in this about not using the word roadmap or why Jonas doesn't like to use the word roadmap, why process can be a dirty word, um, and so on. And I think it really does make a difference. You gotta choose your words wisely cuz it can really decide how excited a team, how enthusiastic team is gonna be about what is a, you know, can be a a, a hard process to, to, to make work. But, you know, maybe the thing I like the most about this conversation, Sean, is just part of the reason why language matters is that Jonas believes that growth is supposed to be fun. Um, you know, he wants that rigor around experimentation of course, but he wants it to result in a culture where people feel like their ideas can come to life and they contribute at all points along the way.
Sean Ellis 00:03:29 Yeah. You know, it's, it's, uh, this balancing act that we always have to, to keep in place where, you know, processes, it's, uh, there, there's a negative, uh, connotation to processes. Yeah. And they can really drag down morale. But I just, I like the way that he thinks about it is he's, he's trying to keep things lightweight. He's trying to stay focused on the goal and, and process should serve reaching that goal. And shouldn't shouldn't be all about the process. It should be all about the goal. And in his case, it really seems to be working. So the business has accelerated. They've got, uh, I think 1.2 million members. And, uh, the really impressive thing to me was the revenue number 2 billion with a b, uh, in revenue. That's, that's, uh, not, not a lot of companies reach those sorts of numbers. Yeah. And so I'm super excited to see what happens as he and his team continue to stoke that fire because they really do have an approach and, and a big mission that tells me that they can do really big things with this business.
Ethan Garr 00:04:26 Yeah, me too. I I can't imagine that over the next few years, like more and more people are gonna become aware of Thrive and excited about what they're doing. It's, it's a, it, it was a cool conversation. It's a cool company. He's definitely a really, um, impressive leader. And I, I really like, I think I, I dunno about you, but I felt like I learned a ton. So, uh, yeah. So should we jump into it?
Sean Ellis 00:04:45 Absolutely. Let's do it. All right. Hey, Jonos, welcome to the Breakout Growth podcast.
Jonas Klink 00:04:59 Thank you. Thanks for having me. I'm excited to be here today. Yeah,
Sean Ellis 00:05:02 We're very excited to have you on. And I'm joined by my co-host Ethan. Hey, Ethan.
Ethan Garr 00:05:06 Hey Sean. Hey, Jonas, good to be with both of you. I'm excited to, I'm excited for this one as always.
Sean Ellis 00:05:11 <laugh>. Absolutely. So, so Jonas, you are the Vice President of Product Management and Design at Thrive Market. And, uh, I'm sure that there's gonna be some from our audience that are familiar with Thrive Market, but for those who aren't, can you give us a quick introduction to the company?
Jonas Klink 00:05:28 Yeah, absolutely. Yeah, so, so the company is still on a startup stage. It's been around since 2014, so relatively mature as a business as it works, right? We've delivered, um, you know, in the online grocery space, we're focused on sustainability and healthy living and bringing, bringing that access really to everyone, not just in the areas where it's already popularized, um, say LA and, and New York and so on. But really important to us is also bringing it to food deserts and, and areas where you have a lot less access to, to healthy and sustainable food options. So the company's has been doing great. Uh, they've delivered about 18 million, um, member orders generated over 2 billion in revenue so far. And, and we've grown now to a size where we're just about 1.2 million, um, members at, at any given point of time.
Sean Ellis 00:06:16 Oh, that's amazing. And, and maybe a little bit more explanation just on, um, so it, it's delivery of groceries or, um, like what's, what's the, what's the actual <laugh> business thing? That's
Jonas Klink 00:06:28 Right. So, so it's, uh, it's a subscription-based online grocery model. So essentially you, you sign up typically for an annual membership, um, which then gives you up to an average of about 30% discounts on, um, all non-GMO, organic, heavily vetted brands, um, that have all been, um, hand selected really, um, by the theme to make sure that they are developed in, in alignment with, with our values, uh, as well as, uh, those of our members. You know, you can look at it as, as a, as an online version of a, of, you know, a given healthy food store like Whole Foods or so on, but with a much broader reach and, and, and effort and even more ways that our members can come in and, and really shop in ways that are aligned with what they're looking to do. So we, we, we supply, um, tools for you to shop by, um, a little bit over 90 different, uh, diets and values. So it's everything from, you know, vegan vegetarianism, keto whole 30 to, um, women owned, bi bi park owned businesses to, um, you know, recyclable packaging, um, regenerative agriculture, whatever have you. Right? So it it, it's really a great space for someone who's passionate, not about just their own health, but also about the health of the planet.
Sean Ellis 00:07:46 So is, uh, is a, a reasonable analogy to say sort of as like a, a cloud kitchen is to a, to a typical restaurant where they're not setting up a physical restaurant. The whole thing is, is for delivery that you're, you're kind of the same in the grocery space where you don't have a physical grocery store, but you have a kind of central distribution center, uh, and and it's all ordering that way, or am I missing something there?
Jonas Klink 00:08:09 Yeah, yeah. So at the, at the end of the day, right, it's, it's a, as I said, a subscription-based model, um, upfront. And then once you're a member, it's essentially a, a, a pretty conventional e-commerce, uh, experience where you build your carts. Um, we have a, a very powerful subscribing save program we call autoship, um, that, uh, is really just meant for, for you to have the convenience of, of shopping at, at your own leisure or, you know, we, we'll send you your favorite products, you know, based on your, your own creation as well as our recommendations, um, on a monthly basis or whatever cadence that, that you determine is right for you and your family.
Sean Ellis 00:08:46 And then do you partner with kind of organic grocery stores, or, or it's, you're, you're sourcing everything and, and like where do, where do the actual, uh, how does the actual like get getting the physical goods to people work?
Jonas Klink 00:09:01 It's, it's a combination. So we have some, some larger recognizable brands, uh, again, in a very much aligned with our values. We have a very rigorous batting process where not only, you know, non-GMO organic and, and these 90 plus different diets and values, but there's also over 500 ingredients that we just never want to see in any of our products. So it's really a combination of, you know, if they're an existing product that we feel meets those, that quality bar, um, we'll set up partnership with them. But in many cases we will source, um, our, our products directly from local farmers and so on as much as possible as well.
Sean Ellis 00:09:38 Okay, cool. I think I've got a handle on it. Ethan <laugh>, uh, any other clarification you wanna, you wanna try to cover? Yeah,
Ethan Garr 00:09:43 Uh, no, Jonas what I was, uh, I wanted to dive into a little bit about your story and how you got here, cuz um, you and I spoke and, uh, it was really fascinating. You've worked for some very interesting companies and some interesting roles, but I, I know you're relatively new to the role, but what was it that got you really excited about applying your experience from those, um, those ex from those roles here? And maybe just give us a little background on, on yourself as well.
Jonas Klink 00:10:08 Yeah, no, for sure. Um, it, it's, it's really just really felt like a homecoming where the kind of 15 plus years that I've spent primarily in the eCommerce industry, uh, all kind of feel like it's led up to this moment for me. So, you know, if I go way back when I, I come from a computer science background, I, I tell my engineering team, I used to do honest work back in the day where I'd actually write code and, and dig into the technology before I moved over to the, to the, to the quote unquote dark side of product management. Um, but, but really, you know, I, I come by ways of starting at the technology company, Google, in, in that case, moving into more technical product management and then, um, got approached, uh, by, by eBay, um, to go work on, on their accessibility program, which is really was really my, my first love.
Jonas Klink 00:10:56 Uh, that's where I did my thesis working with, primarily with those visually impaired, but really across any special access needs. Um, and I was able to go into even apply that skillset into e-commerce and then gradually hand off the reins to the accessibility program and go deeper into the e-commerce side of things. So, so I've had the good fortune of first being able to work for the biggest marketplace on the e-commerce through eBay. Then for the biggest, the biggest retailer, uh, cause I went to Walmart after that by ways of, uh, of jet.com, a startup that they had acquired. Um, and, and really once I get got towards the end of my Walmart tenure, I was really looking for something where I could do something that felt more, um, even more mission driven where I could really get my hands on, on aligning kind of the superpowers.
Jonas Klink 00:11:45 I feel like I bring professionally to the table with doing things, um, that I was also just very, very passionate about. And, and there's two things more than anything, health and fitness, um, especially for myself and my family as well as the environment sustainability. I'm, I, I am that weird guy who will come back from walking the dogs with, uh, a handful of recycling in Sohan, um, cleaning up the neighborhood just naturally. So, so I spent time a couple of years at, at ww Weight Watchers helping stand up their e-commerce best practices. And that's really where I got exposed to, uh, subscription based models as well. And so then when this opportunity came up with Thrive Market, and I, I met with Nick and Saha, the co-founders, uh, this, this just, this just clicked because now everything felt like it had come together where they wanted to bring in someone with a lot of, lot of experience, uh, both within e-commerce and subscription, but also just really to stand up to the product management best practices.
Jonas Klink 00:12:41 And so it gave me an opportunity to just, just come in for the first time, really, truly be able to do product a hundred percent the way that I've always been wanting to do it. Um, I'm, I'm a nerd. I consider product management the craft. I read, I discuss, I, I listen, uh, anything I can do to learn more about product management, and then I warn my team that I, you know, intend to use them as Guinea pigs to, to test out these best practices in, in a, in a practical setting. Um, and so I kind of built my brand of product over the past 15 years and, and as I said, with Thrive Market, it just, just all clicked where I could bring my love of e-commerce and, and more lately subscription together with my love of health and, and, and the planet and, and then also a healthy amount of, of nerdiness around, uh, product management standing it up from scratch here, really.
Ethan Garr 00:13:32 I'm just curious with that, like, was there any, can you point to any, you know, single experience at one of those companies where you're like, I didn't know how valuable it would be at the time, but I'd bring it to work every day now and it it's really helped me.
Jonas Klink 00:13:46 Yeah, I mean, it, it's, um, it's, it's funny when you start working with big companies like this because there's a lot of things that work really, really well, um, on scale. Um, and when you have almost unlimited resources, I mean, for those of you who've worked for big companies for some reason, even if you're the, the world's largest retailer employer at Walmart, it always feels like you don't have enough, enough people to do all the things you wanna do. Right? But I think that one, one of the things that, that I took away was really that, um, there, there's no excuse really. Whatever stage company you are, whatever size you are, there's no excuse not to do things the right way and, and aspire to apply some of the best practices that are out there. I think that a lot of companies will trade, um, being busy and, and producing a lot of activity from really trying to distill things down and looking at the core of a problem and, and undervaluing a a lot to what product management brings to the table in terms of truly understanding that there's a problem here worth solving for your design targets.
Jonas Klink 00:14:54 Um, and I think a lot of companies will just kind of rush headlong in and, and as I said, confuse activity for progress. And, and so that, that's a takeaway that, that I try to play e play every single day. This came up, we just had an all-hands earlier today where people were asking about the classical iron triangle, you know, fast versus cheap versus good. And, and I think that looking at learning velocity and, and really thinking about, um, how do you incrementally and cheaply build confidence around an idea and just kind of graduating it from, from stage to stage and, and only doing as much work as is needed, um, for you to be able to prove or disprove your current hypothesis. That's an art form as much as it is a science. And, and it just, this just makes all of this deliciously fun to me.
Sean Ellis 00:15:46 And I imagine, especially in a, in a company where there is that perception of unlimited resources, the desire to try to get to, uh, validation of a hypothesis with the least amount of work and the least amount of resources, maybe there's a little less pressure to do that. Um, but I know it sounds like, it sounds like, uh, even if there's the perception of unlimited resources, you were still, you were still faced with, uh, resource constraints there. But do, do you find in a, in a startup environment that, uh, try trying to validate hypotheses? It's just more natural to, uh, focus on the, the fastest, cheapest, easiest way to validate a hypothesis?
Jonas Klink 00:16:26 I, I, I found that, I mean, obviously I have a data point of one, right? This is the first kind of true startup because when I was with Jet.com, they'd already been acquired by Walmart. So although it was still kind of a startup like environments in, in our day-to-day work, it was not executing truly as a startup. And, and I think what's been refreshing coming into the startup world is that, uh, there isn't a lot of these preconceived notions that maybe come, you know, with certain companies feeling like we got it figured out. You know, we established product market fit 20 years ago or 70 years ago, and we got this down, and all we need to do is just keep making things better. And as long as we keep making things better at NF scale, we will keep making more money, which is, which is many cases true.
Jonas Klink 00:17:12 Um, but, but it just, it makes the environment kind of stale. And, and when you're part of a smaller company like this where you come in and, and I felt like I had some pretty radical ideas that, you know, Nick and Saha, the co-founders, at least from their perspective, they had never been exposed to it before. They just essentially said, yeah, we hired you because you're, you're the expert in the room, so go do it. Right? And, and that was just incredibly refreshing. Many other big companies, you know, I would have to cut through like three months of red tape to be able to prove why my model might be better than what we had done previously for five years, 20 years. Right.
Sean Ellis 00:17:50 Yeah. I'm, I'm curious on that hiring, um, you have a pretty unique title, uh, being, uh, VP of Product and Design. I'm curious if they were hiring for that or they were hiring for just one, and you said, yeah, guys, it really makes sense to combine 'em both. Like how, how did the decision to call the role of VP of, of product and and design, uh, come about?
Jonas Klink 00:18:13 Um, I think some, some of it out of necessity, uh, and, and some of it out of utter foresight, I I think that there's, there's a lot of overlap between, um, to me the UX design practice and the, and the product management practice. Because at the end of the day, you are trying through different mediums to, you know, essentially advance an idea or advance an hypothesis to, as I said, through various stages of, of, of tests, right? And, and I think bringing the, the product and new team together for, for an organization of our sites made, made a lot of sense. Um, the, the team was reporting into to Sasha, the co-founder, who was also the CTO before. So it was really a necessity, I think, for them to, you know, they both, both wanted to kind of level the role and bring in someone who had done product before, but then also have someone who had the time to really invest in the team and really grow, especially the ways of working, but also, you know, grow the talent into, in, into really its full potential.
Jonas Klink 00:19:13 So I think having the team working really closely together, we're effectively, each of my PMs has a UX counterpart, and, and, and they're paired up as, as a atomic unit almost. Um, and they really look at two sides of the same coin, where typically the, the product person will, will spend more time on the quantitative angle and, and the, the UX person will spend more time on the qualitative angle. They also own doing his research. We don't, we don't have, we're not large enough to have our own use of research standalone team yet, um, which actually has turned out to be pretty good, uh, because it, it avoids kind of creating another team that can often become a, a bottleneck. I think every organization I've been part of has, um, whenever there's a standalone user research team, the inevitable discussion is how can we possibly get more people on the user experience side?
Jonas Klink 00:20:06 So when these things become everyone's responsibility, you know, the, the UX designers in many cases are pulling data as well. Um, on the quantitative side, I think it, it brings much more well-rounded individuals to the table where everyone is, is showing up steep in, in customer insights. Um, and, and it's actually worked really, really well together. I mean, it, it, in full transparency, there were, there was some nervousness on my side because while I have worked really, really closely with design and and UX my entire life, um, it, it's, this is the first really formal, um, experience I've, I've had with truly having the team reporting directly into me. I've had kind of dotted line relationship to the salon before.
Sean Ellis 00:20:49 Sure. Do, do you, uh, have a separate growth organization then, or is that kind of rolled in what you guys are doing? Or how, how does that fit?
Jonas Klink 00:20:57 So we, the way we've got it set up is that we have, um, we have the, the product team that is focused on growth, the two, two of our pods or, or scrum teams, we call them pods, um, are, are leaning into growth pretty much full-time, both precision and retention. And they work very closely with the business side of the house, uh, that are managing more our, our social media campaigns and so on. So they're, they're in short, they're responsible for bringing the eyeballs to the site. We are primarily responsible for turning those eyeballs into, into happy and, and, and paying subscribers. Um, but we work extremely closely, um, on a week to week, almost day-to-day basis to make sure that ideas get cross pollinated and that that the story that is told is consistent for e everyone who's coming in and going through the whole funnel.
Ethan Garr 00:21:45 I just wanted to go back, um, I thought it's interesting when you said each product manager has a UX counterpart and they work very closely together, um, which seems like a really smart way to do things. You mentioned earlier that, um, one of the keys with experimentation is not trying to do everything but trying to focus on the right things, uh, to do. Do you think having those, that tight coupling between design and product allows those teams to kind of slow down and focus on what's important more? Um, do you think that's a big part of that?
Jonas Klink 00:22:15 I think so. Um, we've tried to really kind of in, in the root of it, democratize the ideation. So we have, we have, we have, you know, weekly instances where the whole team is involved, including engineering, QA, analytics, and, and the business side of the house. So ideas come from everywhere, but ultimately the PM and the New York, uh, individuals are responsible for making sure that there's a very healthy I idea funnel that, that flow in. And especially in member growth, where that's a high velocity testing area. Um, and so it, it puts, it puts a good amount of pressure on the team and, and, and on the individuals who are involved in there that they are, you know, both kind of create creatively inclined, but then also really steep and understanding exactly what's happening and, and where, what does the competitive landscape look like? What does our funnel look like?
Jonas Klink 00:23:12 Where is traffic coming? Where, where is it leaking? So it's more of a sea than it is the funnel. Um, and, and that takes a certain kind of mindset and a certain kind of individual. But I think pairing up the oftentimes more creatively inclined, uh, UX individual with the oftentimes more, you know, data inclined, um, GIMs has been, been a really successful combination because to me, the quantitative and qualitative insights go hand in hand, right? The, the quantit insights tell you what's happening, and the qual insights and the designs can help you uncover why that is happening, right? And, and without having both together, you're really just seeing part of the picture, right? So, so to me, th this feels like a very natural pairing and, and it's worked extremely well.
Ethan Garr 00:23:59 Yeah, it seems like a, a good parent also, because it's funny, sh a few weeks ago, Sean and I spoke in front of a very large CPG company, and one of the things, I think in one of the questions that came up, I think, Sean, you mentioned it, you said, you know, often it seems like the two places where you run into bottlenecks with experimentation are data and design. Um, and that's not to say that, you know, those, the people there don't play nicely with others. It's just that those are interesting resources to have to tap into. So it seems like being able to pair those together the way you're doing it, um, might actually sort of unlock a lot of, uh, opportunities. But I, I think one of the key, and one of the reasons for that, I think is that, and I experienced this myself too, is that, um, you know, design is always balancing this, you know, this, this tight walking, this tightrope with, um, get it done fast and do it right, like really, like high quality, you know, do the research, do it right, versus get it out there because the product manager's saying, we wanna run this test next Tuesday.
Ethan Garr 00:25:00 Do you how, like, how do you, how have you looked at that as a whole and managed that? Um, and, and how, what, what do you think is is key to finding, to sort of finding that balance?
Jonas Klink 00:25:11 Yeah, no, that's a great question. Thank you. It's, I think at, at the end of the day, right? It, it, it's, there's a couple of important building blocks here. One is to have a very clearly articulated direction that you're ultimately working towards. Th this is the case where I will actually use the, the expression North star. Uh, you'll find me a bit of a stickler for some of these terms, cuz I feel like they've been overloaded and overused many times. To me, a north star is something you're, you're always working towards just like the North star and, and the night sky. Uh, but you're never actually going to get there. Um, and that's absolutely the case with our, our growth funnels and our growth scenarios where we do need to have a very good sense of where our current hypotheses are taking us. Uh, because otherwise you're kind of just led by the nose, uh, of, of every single ab test and you're, you're operating very in a very reactive manner.
Jonas Klink 00:26:06 So that's one. Two is to make sure that you have a really good foundation to stand on. So part of what the team is working on is to have a, a very clean design system so that the work we do, especially since it's moving very fast, is consistent. Um, but always, of course, as with any design system allowing for extensions of it. So, so the pods everyone is, is, is, is empowered to go and, and suggest new design patterns without slowing down. And then we can think about if it's a, if it's a pattern that should live within the design system, or if it was a one off. And then, and then three, as you're, as you're building these experiences and you're trying to move really fast, one of the things that, that I think is really important that, that I emphasized a lot coming in, was kind of the importance of, of, um, the scientific rigor around AB testing.
Jonas Klink 00:26:55 Because especially, um, in my experience, I love, love all my designer friends, love the ones I've been working with and, and the, on the team. But it, it's very tempting to apply kind of the scout rule when you come in and you're like, oh, you know, I can touch a, B and c, I can move these things around and I can leave this whole thing a better place. And you end up with, with tests unfortunately, that are not clean, right? And so one of the things I had to really kind of emphasize coming in was, we, we are going to have, have, you know, discipline around our testing and we're going to have it be hypothesis driven testing where the hypothesis needs to be clearly articulated. We actually put the hypothesis in directly in Figma as well. So everyone is very clear. Whenever they see a design, there should be a hypothesis right next to it.
Jonas Klink 00:27:43 And then the designs are meant to only touch as much to prove or disprove the hypothesis. Nothing more, nothing less. Right? And, and in some cases, you know, if we, if we think that there are things that are innocent, we might make exceptions, but I've seen a time, and again, through my experience, that whenever you make changes that feel that look very innocent, that's where you get yourself into trouble. And then you end up with 15 changes and you have a test that came back fla or negative in your AB testing, um, and you don't know which of those 15 innocent changes that drove the negativity. And, and I've told the team, at the end of the day, you cannot fail on AB testing. In fact, we classify our ABV test as either a win or a learning. Um, there's no losses, even if they're negative. Uh, the only way you can fail on a testing in my perspective is if you didn't learn anything. Right? And, and, um, so, so I encourage the team to kind of take risk and be bold, but also be disciplined in, in how we execute and, and, uh, and subsequently how we design for these tests. Mm-hmm.
Sean Ellis 00:28:48 <affirmative> any tricks to, to keeping the velocity at a high pace? I, I've in, in my current role, um, I'm, I've been running up against some big projects that, uh, you know, big, big, big tests, but kind of projects behind them that clog up the works a bit. I'm, I'm curious if, uh, if, if you've found workarounds so that, so that you have a, have a consistent release of experiments kind of week to week.
Jonas Klink 00:29:17 Yeah, we, we try to, some of it has to do with, with mindset and just kind of setting the expectation for the team. It's, it's a small team, but we're still, we still set the expectation early on that, that that a week is just about the longest investment we will put on, on most tests because the, the, the hypothesis should be easy enough so that you can, you can run it and, and articulate it very clearly and, and build the test very quickly. And so setting that mindset, I think goes a long way. And, and then we, we, we stick to that and, and because then when we do grooming, when, when the team, when the engineering team is story pointing, some of these tests, if the estimates come back and they're over a week, that just, that triggers a low part of our, our process that essentially then we have a, a broader conversation.
Jonas Klink 00:30:10 That doesn't mean we don't run tests that require more than a week worth of investment, but we do. So, uh, we take that very seriously and, and we make sure that these one week test don't turn into two, three, you know, without any kind of conversation, right? So I think, I think the mindset and then building it into your process that you're, you have the right kind of triggers and alerts whenever you deviate from that process goes, goes a long way. And then also, um, the, the other part is just, you know, really also making sure that we build confidence incrementally. Um, so that another, another kind of latent muscle we had to really practice coming in was that, and this is, again, this goes out to my engineering friends, which I also love. And I worked with tons of, tons of talented people there.
Jonas Klink 00:30:58 But having been in that world myself, you tend to want to build for scale, for security, for all the bells and lists on the first version. And that's where you also have to be really rigorous. And, and that's where the PM needs to be very clear and say for our v1, and I intentionally don't call it mvp, cuz that's another heavily overloaded term, but for you read one, what is it that you're actually trying to learn? Let's not over-engineer it beyond what you're trying to learn, right? Because again, it's all about learning velocity to me and, and don't, don't tell the executives at the team, but the, the wins we get on the business side is a, is a, is a ha happy cherry on top for me. I'm really in it for the learnings because that's what's actually going to advance our, our, our members' experience and, and our backlog and, and constantly defeating new ideas. And then if we move the business forward in the process is great too. Yeah,
Sean Ellis 00:31:52 I think the wins are a function of the learning anyway. If you, the, the more learning you have and the more that you actually capture and understand that learning, the more likely that you're gonna be able to decide on, on experiments that actually provide a pro positive impact to the growth metrics that you're trying to move. And so focusing, focusing only on wins I don't think actually gets you anywhere, but focusing on learning is more likely to result in a lot more wins. So, um, yeah,
Jonas Klink 00:32:21 And also holding the leadership team accountable, um, in, in reverse. So we have a couple of Slack channels to be used internally. One where a product brief gets posted, uh, or in the case of the tasks, just a very simple template with the problem statement hypothesis and proposed solution and the variance that we're going to run. But in, in other cases where the proposal's more complex, it's more of a written out, um, written out product brief. And then we have another channel where we do New York's reviews, where the, the broader team as well as other stake key stakeholders can get tagged and pulled in. I will give feedback on the designs. It's very important, um, that neither of those are blocking channels. Um, so that the, the PM who posts most often to the brief channel and the designer posts to the UX channel that they're empowered to go in and say, okay, I haven't turned anything here for a little bit. We're gonna be grooming today, you know, speak now or forever hold you piece, right? And that's our commitment also as a leadership team, because that's also an area where you can identify bottlenecks, right? And, and, and easily become a, oh, you know, you need to go through the peanut gallery to get something out there. Say, no, we hired smart people. Um, we've empowered them and we, if we don't make the time to review the work, we're just gonna have to let them do their work. Right?
Ethan Garr 00:33:42 So I'm curious, uh, you, you really, uh, you touched upon the testing rigor, which I think is super important. And I think, you know, I think Sean's question, uh, also alludes to the fact that sometime earlier question that sometimes it's very hard to stick to that rigor. Sometimes there is that second, uh, there's that one little thing you want to include, you don't wanna wait for. Um, and I think that those, those are always the, the real life challenges you have to, you have to deal with when it comes to experimentation. But I'm curious, um, in terms of when you say, you know, when you talk about, um, this commitment to hypothesis and, and the only failed test is the one that you don't get a learning on, has there been one or two common reason? Have you had those failures where you didn't get the learning that you say it's usually because we didn't do this?
Jonas Klink 00:34:31 Yeah, yeah. That, that happened, um, in one of the product areas quite frequently in, in the first quarter, actually. And, uh, it really, the failure to learn just came from essentially that we, we felt like we had some pretty well-informed hypothesis around customer pain points. Um, where, in know, in full transparency as we were trying to overcome the Cold Start problem with that team's backlog, uh, we felt okay going in and following industry best practices and focusing, um, testing, um, of those industry best practices, an area of high reach that we were going to be able to supplement, um, going to, to do the work which we were doing in parallel to truly identify customer pain points and, and go after those opportunities. That is not a mistake I will ever make again. Um, <laugh> because it, most of the tests essentially just came back, came back flat, and so that didn't mean technically they wouldn't learn anything as, as an aggregate, we learned that we were operating in areas where we thought there were pain points for customers because they, they're well-known pain points in the rest of the e-commerce world.
Jonas Klink 00:35:42 It just happened to be that it just didn't apply at the same scale to the Thrive markets members that were applying it to. So I think in those cases, the, the meta learning from that was that we, a, we need to conclude our, our customer research. You really need to, to drive with insights. And so I won't make that mistake again. I, I thought I could outsmart myself. Um, and b, we need to make bigger swings, right? We need to go and, and really provoke a signal and, and the guidance I've given to the team heading into this second quarter is let's make sure that we have a few tests that I don't care if they are 10% positive or 10% negative, but let's get out of this kind of flatlining that we've been operating in. Um, because it just, it just sends a strong signal to me that we haven't yet to really uncover what our customers truly care about when it comes to this experience. Right. And, and there was such a, such a fantastic insight and, and, you know, a humbling learning opportunity for me in the process as well
Ethan Garr 00:36:40 With, with that commitment to trying to keep tests to under a week. Do you find that those sometimes run, run against each other where it's like, Hey, we want to take bigger swings, um, but like bigger swings sometimes require, sometimes do require more investment in resources and, and things to, to launch? Or do you feel like you're able to to, to control that?
Jonas Klink 00:37:01 Yeah, no, so it's a, it's a good clarification. So the one week is, is in one of our product areas, the, the, the growth product area. In other areas we, we operate kind of on a different timeframe, but it's not, it's not, it's not a hard and fast rule. I think we've seen enough in our, in our growth areas that, um, when it comes to pure member growth, it, it's been velocity, um, has been the name of the game, uh, while of course keeping a very, very close eye on the wind rate. And so, uh, we've been able to maintain a very healthy wind rate in that area. So it's really just, it's, it's been a function of just how can we just do more, more tests, right? And, and push more through the pipeline because of just, just simple math, right? And we keep the wind rate than the average wind going, then just more tests will just get us more wins, right?
Jonas Klink 00:37:50 But that is not true in every product area. The other product area that I was just talking about in terms of, you know, a lot of the tests being flat, we were initially experimenting with a slightly lesser velocity version of that same strategy. And, and it just, it, it's just not going to be the strategy. We could, we, um, we, uh, continue with into Q2 because we're just, we're just going to need to take bigger swings and it's going to take a little bit more time. So it, it's, there's no one size fits all here. I think, uh, it really comes down to the nature of the problems. You have some areas where it's, a lot of it is about eliminating friction like in, in growth funnels and so on. Velocity is very helpful. Are there areas where you're still trying to understand, um, what the member opportunities, the pain points look like? Velocity is not gonna be your friend because then you're just running headlong into generating a lot of tests that come back flat, right?
Sean Ellis 00:38:46 Yeah. So a coup couple of questions for you here. Um, first, you've, you've referred to the team as being relatively small, but you've been on some massive teams, <laugh> between Walmart and Google, and probably Weight Watchers as well. So, um, at least relative to, you know, early stage startups. So I'm curious, when you say the team is small, how, how many people are actually on, uh, like employed by the company?
Jonas Klink 00:39:11 Uh, so, so, so on the, on the corporate side, it, it's, it's a couple of hundred in, in total. Uh, it's, it's uh, a little bit over, um, a thousand or a couple of thousand. The, the product and UX team is, uh, very small. Uh, we're by my measures, uh, we're five and five, so five, five PMs and five UX designers.
Sean Ellis 00:39:33 Okay, gotcha. Yeah. So it's just good to good to get that kind of relative scope. Cuz I did see yeah, on, uh, on LinkedIn a number that was north of 500. And so, um, but, but again, there's probably a lot of logistics people that might not be on, on LinkedIn or part of the logistics, uh, process there that might not be on LinkedIn. So Good, good to get that. Then the second piece is, with so much focus on learning and velocity, how do you keep track of all that learning? How do you, how do you make sure that the right people get access to the right learning? And as that repository of learning grows over time, how, how do you actually, uh, make it so that you can navigate and, and you're getting smarter over, over time rather than just accumulating learning that then no one accesses?
Jonas Klink 00:40:17 Yeah, yeah, it, it's, it's, it's a great question and, and it's really where kind of process a waste of working. I usually call it waste of working cuz I find that a lot of people, you say process and it's almost like a swear word in some, some circuit site. I, I think, I think processes as an aside has gotten a bit of a bad rep because I, I'm, I'm a firm believer that as humans we tend to kind of organize in some way or the other. And so whenever you come to a place where it says you, we don't have any process cause we don't like it and we don't need it, there's always a process. It's just a matter of whether you wanna be in, in control of your process or not, right? Cause it's just human nature. And so I think in this cases we've tried to really identify what is the lightest weights touch kind of, of, of this process to democratize the insights because I, I, it's a great call out.
Jonas Klink 00:41:05 I, I think that if people are going to contribute ideas, they need to have access to the insights, then they need to have access to it in a way that doesn't require, you know, attending hours of hours of meetings. So e each, each of our, each of our groups, they will stand out a, a summary on, on Mondays and just says, you know, this is, these are the tests that we're running, this is what we learned last week, here's what's gonna go into, into testing this week. And here's what we're doing against our on ongoing accountability metrics. So very lightweight, very simple template, um, but it gives people everything they need. And we link into our, our IB testing platform and so on for people who want to go go deeper in there. Then we do monthly check-ins with the leadership team on every team level where they really kind of just roll up all the insights, um, and, and share what's, what's on the, on the roadmap they have. And then the working team will operate more on the weekly basis where the whole team get together and, and they will do rice scoring together of, um, the ideas backlog. And so everyone should have, by the time an idea actually makes it into the hands of an engineer, everyone should have seen every idea at least, at least once, if not multiple mm-hmm. <affirmative> times. Mm-hmm. <affirmative>.
Sean Ellis 00:42:17 Yeah. So I, you know, one of the things that I am struggling with myself there recently is kind of, there's learnings on the experiment level that, uh, makes sense to attach to the experiments, but then what's, what's the, what's the high-level learnings that kind of connect all the things together so that you start to see, we've run a bunch of experiments in this area and none of them have really yielded great results while experiments in this area always seem to yield good results. And, and then there's a lot in between there. Uh, so I, I don't think I've necessarily cracked it yet, but I'm, I'm trying to, trying to surface learnings in such a way that you, you start to see patterns and signals to help guide future experimentation. Um, but yeah, do you have anything like that, that, that, uh, kind of starts to give you the high-level patterns?
Jonas Klink 00:43:08 Yeah, no, absolutely. And, and you know, we, when we do the, the all hands we do quarterly is a good example where that, that's really a good cadence. I think as a product team where you should, you should take, it's good forcing function for you to take a step back and, and take a look at your strategy documents and see was this something we learned that that, you know, should have us pivoting as we're heading into a new quarter, right? Or did we learn something as fundamental as do we, do we even need to update our vision or, or, and our north star that we're working towards, right? That those should be more rare, but, but they do happen, right? So I think doing that on a periodic basis, I think once a quarter on, on a more strategic level, if it, if it causes you to pivot how, what you're focusing on and how you're executing, which, and I ga I gave that example around the, the flat tests and, um, um, then in general just being very open, uh, with yourselves, right? That every single test can, can change the, the backlog. And, and I think as an aside, that was also a learning experience for the team coming in. Um, because I've also tried to eradicate the term roadmap, um, which I'm also, I know I come across as very, very much a stickler for terms, but I find that when people talk about roadmaps, they expect that, okay, everything is, it's like predetermined,
Sean Ellis 00:44:27 It's
Jonas Klink 00:44:27 Like a waterfall concept, right? Yeah. Where everything's figured out. And, and I said, look, yeah, I can come in, I can tell you what I think we're gonna do for the next three months, but I'm very likely to come back next week and I will tell you a whole different story. So, you know, I'm happy to have that conversation, but I don't think it's good use of anyone's time. So we've, we talk about backlogs and people are, you know, it's a shocking concept, you know, getting accustomed to some of the fundamental agile principles that there's a backlog and, and everything is subject to change if we learn something that in invalidates our previous hypothesis. Right?
Sean Ellis 00:45:01 Yeah. One more quick question, Ethan, and then, uh, then I'll let you take us home, <laugh>. I'll be all out of questions at that point. Of course. I, I say that in my roadmap of questions. There'll be probably six more that'll come from your answer. But, um, the, when you, when you talk about scoring ideas in the backlog, uh, how much, how much sort of pre-ex experiment data analysis are you doing to help guide some of that scoring? Are you doing any, or, uh, like at what point is it sort of paralysis by analysis, doing too much data analysis versus, versus just like, oh, the test is gonna give us the most important data? So, uh, I'm curious how, how you, how you think through that.
Jonas Klink 00:45:41 Yeah, no. So, so, so we use, we use the right scoring framework, right? So reach multiplied by impact, multiplied by by confidence, divided by by effort. And, and I like that there's many different prioritization frameworks. They all have merits, but I like it because it, it makes in a confidence score, um, which really goes hand in hand with your, your impact estimate. So really the way we do it is that whenever someone submits an idea, and as I said, it could be anyone within the company, we encourage people to give as much contents as they have, right? Some people will just submit an idea, move onliner, some people will include competitive examples and data and whatever have you, right? And that just really then shows up when we do the rice scoring, it shows up in your confidence score. And, um, so we, we try to make sure that we don't place a, a very strict onus on, on, because we want the ideas, right?
Jonas Klink 00:46:34 We don't need perfect ideas, we need ideas. And then when we do the rice scoring, if it turns out that it's something that has a high reach times impact product, but maybe low confidence, um, then that's a good opportunity to say, well, um, this idea is not ready to, to go into the next stage for engineering yet, but this has potential. So let's go off and do some more research, look at competitive examples, gather some more data to try to bring our confidence score up and see if the reach and impact still holds, right? So it's really about, in that case, just, you know, you see it, you see the promise of the idea, and you just graduated to the next tiny more step forward of, of additional investment, right? And that's how we're really think about the, the life of an idea for everything we do, that it's really just about having the, an idea jump through various gates. And if it keeps surviving, you move it on to the next side. But otherwise, you should be really ruthless to your own ideas as well, and say, look, this just doesn't hold water. Let's not waste our time here. Let's put it back on the backlog of whatever we need to, to see if it might mature for a later stage and, and move on to the next idea. Yeah. All right. Ethan,
Ethan Garr 00:47:43 You're up. Well, uh, Sean thinks I'm gonna hit us with our final question, but I actually, there's one question I wanted to, uh, to ask you about. Um, we'll have to keep it brief, but it was something you and I chatted about a little bit. Um, and I just thought it was, it was super interesting for our audience, but, um, you mentioned your background, I think at eBay, uh, with where you're, uh, really focused on accessibility, um, which is something that most companies don't have a team for. They don't have anyone des, you know, dedicated to it. In fact, I know at company as I worked at, it was an afterthought when we were at risk of violating some, some, uh, some code or something. Um, but accessibility and design is really important and, and interesting. And I just, I, I wanted to get your, just, if you could just tell our audience just a little bit about why that was important in your own personal growth and what it's taught you and how you use those learnings today. Um, just if you'd give us a minute of that, I thought it'd be interesting before we do give you our wrap up question.
Jonas Klink 00:48:40 Sure, sure. Yeah, no, I'd be happy to. Yeah. And so, so kind of the passion was really awakened. Um, I, I was reading some research papers in the early stages of accessibility product area, and I actually ended up picking it as my, my thesis topic. So I was working with kids K through six that had at least one special access need, if not multiple, so blind or deaf, blind, whatever have you. And, and just seeing the challenges that they were presented with, with the, but the, the, the just, uh, the positive outlook that they had on life and how they took on challenges, I was hooked, right? And that's just been reinforced throughout my career as I worked on accessibility both at Google and, and at eBay. I handed off reigns there and really, um, you know, now serve more as kind of an executive sponsor or, or mentor for the accessibility program that inevitably will be made up by a vastly understaffed group of very passionate individuals that are trying to roll a massive boulder uphill, right?
Jonas Klink 00:49:35 Because it usually just comes down to knowledge, right? Most companies do not intentionally make it hard for people, you know, who rely on screen reduce and what have you to access their tools. It's just that they don't know that they're breaking all these things, right? So I've spent a lot of my time, uh, and, and really trying to figure out processes where you can balance the investment, because the, the flip side of this is yes, it's the right thing to do. Yes, there are very real legal requirements around it. Um, but, but at the end of the day, this is not a growth area where, you know, if you're looking to add another million of users, accessibility is not going to be the path. So how do you make sure that you can still do the right thing? Because it's so important to do, because it's disproportionately impactful to each of these individuals, but you, you right size the investments so that it's actually something that the company can take on, right? Because if you ask for too much, oftentimes at some point you'll, you'll get the hand and say like, oh, this is getting a bit too expensive and we have real o OKRs that we need to deliver on, so please, please go away and don't talk about this for a while. Right.
Ethan Garr 00:50:42 Very cool. Well, thank you for that. Uh, all right, Sean, should, uh, you wanna, would you like to ask the, our final question today?
Sean Ellis 00:50:48 Sure. Happy to. So, uh, Jonas the, uh, when you think about, particularly at Thrive, but you can, you can take a bit back further back if you want to, cuz I know it's been less than a year you've been there. Um, when you look back at least over the last few years, what, what's the thing that you feel like you understand about growth now better than, than maybe you understood a few years ago? Like one, one thing that's a really important understanding that you've developed.
Jonas Klink 00:51:13 Yeah, I, I think that if I were to say the one thing, um, and we talked a little bit about this and that is, that is really just that velocity matters. I think that I, ideas, um, are, I wouldn't say they're easy, um, cause the, the team works really hard, but ideas are plentiful and they're tons of examples of, of lots of competitors that do all kinds of stuff, right? So getting ideas into the backlog is, is not quite as hard, at least we haven't come up on it yet. But being able to execute with velocity and, and maintain the quality of those ideas so that you maintain the win rate. I, I wouldn't say that, you know, that we have it figured out. I certainly not, and I'm, I'm still fairly new to the game, but just the simplicity of, of thinking about it that way, that you're looking at, you know, velocity times your win rate, time to average impact.
Jonas Klink 00:52:05 And if you just keep doing that and you're maintaining, especially an eye on that win rate, you will drive growth, right? Um, and in some cases you need to kind of poke the bear and you need to do something to force yourself out of a, a local maxima. So you're not just doing small, incremental things, but you're also willing to make some big leaps and try something radical from time to time. But I think that that's been one of the most powerful insights to me that, you know, this, this has really been transformational at, at Thrive. To, to move and go from a velocity that was maybe, you know, in the single digits for a quarter and, and infusing talent and, and, and people, and, and being able to test, you know, dozens and, and you know, hopefully even more in the future of ideas and, and continuing to see a consistent win right there. It's been, it's been so simple and so powerful. I
Sean Ellis 00:52:54 Love it. I love that formula too. Velocity times win rate, times average, uh, impact gives you your, your growth rate. Is that, is that something that you have picked up somewhere else? Or did you just come up with that now? Or <laugh>
Jonas Klink 00:53:06 No, no, no. I can't credit myself for that. That's, uh, that that comes from, from my, so many of my readings, so, okay. Yeah.
Sean Ellis 00:53:13 But
Jonas Klink 00:53:14 One of those things where I was like, this makes so much sense, I can't wait to try it. Yeah. And I tried it and, and, and, you know, I've been fortunate the company's been been open to just kind of run with a lot of these, you know, semi crazy ideas on the surface maybe.
Sean Ellis 00:53:27 Yeah, I've, I've often said, you know, your success is a function of your, uh, quantity of ideas and your quality of ideas, but that this is way more specific with that formula. So <laugh>, I love that. Um, we'll have to, we'll have to track who, who, uh, who first mentioned that one, but, um, it's been super exciting having you on the show today. And, uh, I don't know, Ethan, you have some, some key takeaways you wanna share?
Ethan Garr 00:53:53 Well, I think it goes to Jonas. I think you've really brought a lot of, you know, shined a really good light on just the importance of scientific rigor. Um, but I think you've also pointed out that within that scientific rigor, there is the reality of every day, and you have to deal with, with, with those realities. Um, but I think the attention to the scientific re uh, rigor is what, what really drives that, that long-term success. So that was helpful for me. And, um, uh, while I think there's no one size fits all, as you've said, and, and that's always, you know, you always have to adapt your growth process to the situation you're in. Um, I do think you, you've created, you know, you've shared with us some really sort of pr, you know, important principles that should underpin every process. So, um, yeah, there's been super helpful and I, I've really enjoyed it.
Sean Ellis 00:54:40 Yeah. And I also think, um, you're, that being careful with the words that you use, um, is, is interesting. It's important to be careful with the words that you use. Like for example, when you talked about, we don't talk about winners and losers, it's, it's, it's winners and, and learnings. Um, but, but even with your, your hesitancy around process is a word because process sounds heavy and can slow you down. I do. Yeah. I break my learnings into customer learnings and process learnings. When I, I, we just had our, our monthly meeting and that's, that was the organization structure that I, that I used. And, um, but I do think, I think process is important here and, um, and the benefit of process is that when you document it and you think about ways to improve it, particularly around how do we, around that formula that you laid out, you know, how how do we improve that velocity? Velocity is the thing you can control the most there, the win rate and the average impact is a lot harder to control, but at least you can control the, the velocity and, and hopefully over time you're getting smarter with the win-win rate and average impact. So, um, a lot, I think a lot of really good, really a lot of really good things that you've talked about here today. And, uh, I'm, I'm excited personally to go back and re-listen to it, so thank you so much, Jonas. Any, any last words before we wrap up?
Jonas Klink 00:56:01 No, I, I think also what what's been gratifying with this process that I, that we talked about here is that it's just kind of faded into the rafters into the background. And, and I can, I can tell the team is having a lot of fun, which to me is, is what it's all about at the end of the day, right? I always kind of look at things as we, we take our work seriously, but ourselves less though. And, and the team has taken so well to this very kind of high velocity that could be stressful and, and, and a lot of pressure in the process. It's an important company goal. Uh, but, but that's just there and it just, it just happens. And, and I think that's a testament to, to the team and, and how well this is working and they're just having fun. They're talking about the ideas and the customer problems, and that's how they're spending their time. They're never talking about, oh, you know, what are we doing next and how are we gonna jump this through the different steps that, that just happens, right? Which is, which is extremely gratifying to see.
Sean Ellis 00:56:54 Yeah. And having the worthy mission that you do, I think, um, also helps to protect against that burnout. So keeping it fun, keeping the velocity of learning up, but then having a, a really strong why, um, those two things together. I, I think, uh, keep, keep people engaged and excited. So it's a, it's a fun area that we, we all work in and, uh, it's, it's important to, uh, not have it be as stressful as it can be and, and have it be enjoyable. So, um, it's, it's great to see you doing that. And, um, I'm, I'm personally looking forward to seeing you get a lot more success because it, it, it, it seems what you're working on is super important. So thank you for, thank you for sharing the story with us today.
Jonas Klink 00:57:36 Yeah. Appreciate it. Thanks for having me. All right. Thanks a lot.
Sean Ellis 00:57:41 And, uh, for everyone tuning in, thanks for listening.
Announcer: 00:57:48 Thanks for listening to the Breakout Growth Podcast. Please take a moment to leave us a review on your favorite podcast platform. And while you're at it, subscribe so you never miss a show. Until next week.
