Launch Clinic

The Role of Beta in Product Launch

Software companies traditionally use a Beta program to expose the product to more rigorous testing by customers. Depending on the nature of the product it may be thousands of beta testers or in the case of enterprise software it might be a more tightly controlled number.

What’s often overlooked is the valuable market feedback that should be obtained and acted upon during a Beta program. That’s why Beta programs should be co-managed by development and (product) marketing. Each group brings a different perspective to the party that can ultimately result in a better solution that will generate more momentum at launch.

The Problem with Beta Programs

The problem with beta programs is that most are half-baked excuses for beta programs and here’s why:

• The size of the beta testing population is not large enough to get any meaningful feedback
• The duration of the beta program is arbitrary and too short, lacking specific milestones
• The beta program is run by Development and Marketing is left on the sidelines
• Gathering feedback isn’t a proactive process

It goes something like this. Development is pressured to get the product finished. Company practice is to run a beta program. Afterall, if a beta program is conducted then Development is off the hook because real customers used the product before is was released. Three customers actually give feedback and they are customers that have been using the product for years and tell you what you want to hear. The Beta program runs for the minimum required 4 weeks. The product ships and the larger population of customers complain that the features suck and don’t make any sense.

The Purpose of a Beta Program

The purpose of a Beta program is to get early market feedback on your product. It does not relieve you from the responsibility of doing a quality testing job. From a testing perspective the Beta program gives you an opportunity to test your product in “real world” environments and identify problems that can be addressed before you ship. The real value in a Beta program is to get customer feedback on the usability of the product, performance, perception of value and usage patterns. All of which help refine the message, positioning and pricing.

Running a successful beta program

A successful beta test program accomplishes three goals.  The first is to get greater test coverage.  The second is to get market feedback.  The third is to prepare your organization to support the new product. All of these goals should be considered when planning a beta program and are equally important.

From the engineer’s perspective the beta is a way to move the product into the hands of potential customers who will use the product in wide range of environmental conditions that could not be simulated in the lab.  But a beta program should never be a substitute for good testing practices or usability testing.  You still have a responsibility to provide the best testing you can and the UI should be baked.  It doesn’t have to be perfect, but ship a buggy beta you may have a very difficult time finding beta customers for the next release.

From the marketer’s perspective the beta is a way to get customer feedback of what is liked about the product and what needs to be improved.  It’s also a way to validate market research and assumptions about positioning.  Funny things happen when a product gets into the hands of customers.  New uses for your product will be discovered and some things you thought were really cool (and differential) will get lukewarm response.  Be prepared and be open minded.

From a customer support perspective a beta is a way to interact with customers in a relatively forgiving environment to gain experience handling common issues.  Often developers want to be the ones with direct contact with beta customers.  The upside is direct feedback.  The downside is once the product ships those customers will know who you are and may expect to reach you directly when they have problems.  Using customer support as a first line of defense enables the development team to be more effective in the long run.

Beta tips:

• Don’t use a beta program as a substitute for good testing practices
• Involve development, QA, customer support and marketing
• Have someone with a good market understanding take the lead
• Don’t flood beta customers with new releases
• The beta program will always take much longer than you think it will
• Require registration for all beta customers
• Give beta customers something for their efforts
• Reach out to customers for feedback, including making phone calls
• No feedback = failed beta

Other articles on beta programs:

• Joel on Software - Twelve Tips on Running a Beta Test
• My MicroISV - Running an Effective Beta Program
• Pragmatic Marketing - Building a Better Beta
• Project Perfect - How to Run an Effective Beta Program