The Lean Startup method strongly advocates experiments -- and for good reason. It's critically important for a startup to acquire validated learning as quickly as possible. How quickly can you get through a learning cycle? How efficiently can you get to the answers to crucial questions?
You might run experiments that will answer some of your most pressing questions:
1. Will adding this feature cause more people to start paying for the product?
2. If we increase our prices, will our overall revenue increase or decrease?
3. If we make this feature that was previously free part of our premium offering, will users be upset?
Experiments are great -- but one word of warning. Be mindful of how much data you need and how "clean" your experiment needs to be in order to yield the learning you are seeking. A mistake we often make is looking at the "early evidence" from a particular experiment -- and then, in the interests of time and/or money (both of which are in short supply), use that early evidence to make an "educated guess" and move on.
This "educated guess" based on some early evidence is often "good enough". There are lots of questions for which you don't need perfect answers. All you need is something reasonably better than random -- or something that validates a strong "instinct" you already had.
But, be careful. The rigor of your experiment should match the importance of the issue at hand. If it's a big, important decision that will shape your company for a long time, don't just rely on the "early evidence" and use it to rationalize whatever it is that you wanted to do in the first place. Take the time to let the experiment run its course. For big, important, critical issues -- the extra rigor is worth it.
Example: You want to know whether taking a particular feature *out* of your product is going to have a major impact on your users. The feature didn't work out as well as you had hoped, and it ended up being very expensive to maintain. So, you send a survey out to your 5,000 users. Of the first 500 responses that come back, 80% of the people ranked the feature as "Super-duper important, if you take it out, I'll use another product". So, you could just take this early evidence, extrapolate and say -- "Hey, if 80% of our users really want this feature, we should just keep it in." In reality, what might be happening here is that the users that were most passionate about the feature, and thought that you might cut it are the ones that first responded to the survey. Users that were kind of "meh" (or didn't even know the feature was there) might take a while to respond, if it all. Basically, the early responses are not representative of your overall user-base. If you let more of the evidence come in, you might find that the actual number of users that care is much smaller than the "early evidence" showed.
The Danger of the Self-Fulfilling Prophecy
Another thing to be careful of when it comes to "early evidence". If this early evidence leaks into the organization, you often will trigger a self-fulfilling prophecy and wind up with a potentially misguided decision.
Example: You ask your sales team to start selling a new offer (could be a feature/product/promotion). Understandably, the first few attempts don't work out very well -- the sales team hasn't quite figured out yet how to position the offering. It will likely take a few weeks. In the meantime, word starts to spread that this "new thing" isn't selling all that well. As a result, the team pulls back a bit and reverts to selling the "old thing" (change is hard). This of course, causes even fewer sales of the new thing -- and it ultimately gets abandoned. Now, that might have been the right decision. Perhaps the early evidence was right -- but you don't know for sure. What if just a couple of weeks of training and tweaking would have fixed the issue. Perhaps it would have been awesome.
In summary: Don't confuse early evidence with compelling evidence. Avoid letting early results of an experiment taint the rest of the experiment. And, match the rigor of your experiment to the importance of the decision on hand.
Any examples you can think of when early evidence is misleading?