The De-Evolution of Beta Testing

I’m not going to drag you through a regurgitation of how I got into computing or when, but it was years ago.  My nephew Mike nudged me in the direction and I just fell in neck-deep after that.  De-evolution, or DeVo (haw haw), is the degradative effect of devolving from one state to a lesser state.  That’s what I’ve seen happen with software product beta testing.  I really thought about this after reading Robert McLaws’ recent blog post about his resentment towards something Steve Ballmer said about Vista beta testers.  I agree with Robert 100% by the way.

Along the way I became engrossed in software programming and after college got deeper into software architecture, design and engineering.  But rather than getting lost in the acronymn soup of SDLC, CMMI, and so on, I tried to stick to just doing it right and having fun along the way.  If it’s not fun, after all, you’re wasting your time.

Somehow I got into volunteering to help beta test products for various companies.  It was exciting and I did my best to run their products as hard as I could and exercise all the demons from them I could.  That started in the late 1990’s.  My last involvement with beta testing, in the classic sense, was 2007.  Some companies can be named.  Others cannot, due to legal restrictions, but regardless, they were for the most part large corporate businesses, not small shops.

1998-2001

Primary interaction was by e-mail.  Software was sent via postal mail in discrete packaging, with an NDA insert to sign and mail back (for some, not all).  Sometimes a web site was setup with a secure login to provide and track feedback.  Initial involvement was at the “alpha” stage.  Usually this included a hundred or so testers, prior to progressing to “beta” and including thousands more.  Problem reports, even with respect to documentation, were followed up with e-mails and phone calls (depending upon severity and “exposure” to mass impact).  Enhancement requests were normally given due consideration with sufficient interaction via e-mail to rationalize the effort and prioritization.

2002-2004

Primary interaction was by e-mail and web portals.  Software was downloaded from FTP servers.  Web portals used about as much as e-mail for channeling comments, resolving issues and fielding requests.  Initial involvement was still “alpha” but increasingly not until “beta”.  Initial round of testers was in the 500-1000 range on average.  Problem reports were followed up with e-mails and only closed when resolved.  “Beta” stage still considered enhancement requests as long as level of effort to address was determined to be “minor”.

2005-2007

Primary interaction was by web portal. E-mail being rare.  Documentation efforts were separated from the product trials.  Software was downloaded from web portals.  Initial involvement was overwhelmingly “beta”.  Enhancement requests firmly rejected as being “too late” in the cycle.  As best as I could determine, only largest (highest paying, highest visibility) customers and partners were invited to be in the “alpha” and were given priority channels for development feedback.  “Feature Frozen” at “beta” became the standard.

Side-by-Side Comparison and Contrast

1998-2001	2002-2004	2005-2007
Initial: Alpha	Initial: Beta	Initial: CTP
E-mail	E-mail/Web	Web
In Flux	Rationalized	Feature Frozen
6-8 months	8-12 months	1 year
Product & Docs	Product & Docs	Product Only
Some Marketing	Little Marketing	No Marketing
Voice Comm	E-mail Comm	Web Forum
Alpha: Enthusiasts	Alpha: Developers	Alpha: High Profile
Rewards: Free Software, Swag, Mentions	Rewards: Free Software, Swag	Reward: Free Software
Invite: Ask	Invite: Sales Rep	Invite: Inside Contact

The gradually emerging trend is somewhat apparent, but not overtly obvious: The testing process is becoming more rigidly controlled from all aspects.  This includes who gets invited.  How they can communicate with the vendor.  And what happens as a result of the feedback.  It’s also becoming more controlled from a cost standpoint.  Less is being “given away” and the communication infrastructure itself is being moved from a human/labor base to a web/automation base.  Metrics are king.

After 2007 I enjoyed a career change (that’s a joke, actually) and found little time to devote to participating heavily in testing trials.  I still get invites, but not nearly as often or as many, but that’s to be expected.  My last significant involvement concluded with a simple “thank you” and no offer of a free license or anything else.  Kind of a smack in the face for all the hours spent helping them do essentially what their own (paid) staff should have done.  As I said, I can’t (and won’t) name names, but it wasn’t just one company, and it wasn’t just me.  Having enjoyed the company of many other testers, we shared information about our experiences quite often, so I know what I’m saying is fairly reliable and consistent.

I hope this trend turns around.  It’s really becoming sterile and detached from emotion.  And emotion is what humans base all decisions on, regardless of how logical they should be.  “gut feeling” is always involved.  If you remove “gut feeling” from the perception of your product it becomes just a “product” and not the culmination of all the imagination, the personal involvement, the work and the (oh my God, not the following word?!….) “craftsmanship” that sets it apart from all the other “products” that do essentially the same thing. 

So, Mr/Ms CEO, I ask you: Is that what you really want?!