Podcast Content

Darius: What's the best way to design experiments?

Ken: I think there are two things that are really important in that. Number one is that you have to go into the process of the experiment by knowing what your assumptions are. What are you assuming to be true that there is no evidence in place for yet? So what evidence do you need in order to assume that something is true and improve that out? Darius: So like the Success factors or the metrics that you are going to measure again? Ken: Well, that's kind of the second piece is evaluating that. So I think there are two pieces. One is and this is really difficult, especially in the enterprise, enterprises are made up of experts at something. Dover being a manufacturing company, we are experts at delivering manufacturing and building things and we have certain products that we build, we deliver to the market. We are extremely successful and make a lot of money on them. And things are looking very good in the stock price is great. And that leaves everybody with this feeling of we are experts at certain things. But what happens is, when you start doing innovation, you start wandering into things that are more uncertain, your expertise becomes a problem. It creates a cognitive bias where it creates a series of cognitive biases, but one of which is well known as confirmation bias which means we are always looking for the answers that prove the way we already think! So step one of the designing an experiment process is understanding what you already believe to be true because of your expert status and then set out to prove that that's not true. If you set out to prove that it is true, the expert confirmation bias will take over. There is no need because you will always come back with the same answer you thought you would have when you went in. You'll say to be constantly fighting against that. Or you'll be rationalizing away what you've heard and so that's kind of going back. So step number one is we have to understand what it is that we're trying to determine whether it's true or false because it's an assumption we set out to prove that it's false. And then if we can't prove that it's false and it may be true. So we're gonna work on it. That said, understanding what your assumptions are is very, very important. The second piece of it is something that you then alluded to, which is what are your metrics to show that you have learned what you needed to learn and those are very individualized. The metric should be tied directly to the assumption itself. They're not broad metrics, they're specific and they last only as long as that assumption laughs. And then you need to set more. This becomes really difficult because people don't like to continually reset metrics. For instance, let me use this one at a friend of mine one time and tell me I asked about what kind of success metrics are you using in your innovation program and you started doing that? Well, one of our success metrics is a number of experiments that we run. In my view, it's not the number of experiments that you're running because what that leads to is people wanting to cycle time experiences as much as possible in order to hit a number goal as opposed to a learning goal. So the metrics themselves then and part two have to be tied to what you're trying to learn. Let me give you an example. I had a team I was working with one time. I had this great idea for I won't go into it, but had a great idea for cruise lines. And they were gonna pitch this to the cruise lines because this was going to make the passengers really, really happy. And believe it or not, I think it actually had some merit at the time. It was really kind of. So I'm like, okay, well, let's. You've worked up your value proposition, your solution statement, some assumptions about what you think the passengers would think and whatnot. So and we came to the voices. Our next step was to find out, you know, some of them whether their value proposition resonated with the cruise lines. So what type of metrics do you put in place to find that out? Well, they were thinking about it. The first validation of the metric of your idea is will they even give you a meeting? You have to pitch them something that will entice them to give you a meeting, whether it's either a phone call or a face to face. If you have five target cruise lines that you really would like to talk to but can't get one of those to give you a meeting, either face to face or on the phone within the next 14 days, your idea is dead or you've got to pivot your value proposition, This is a signal your solution or vision is not meaningful enough for the target customer to care.