OnStartups

This blog’s audience can be simplistically divided into two types of people: 

1.  technology geeks (folks with a technology background, and more specifically a software development background) that have an interest in the business issues because they’ve founded or are thinking of kicking off a startup.

2.  business geeks (folks with a business/sales/strategy background) that have an interest in technology because they’ve founded a software startup.  For more on my thoughts on business geeks, read “Business Geek:  Not An Oxymoron”

A number of my articles address one group or the other (like my “Presentation Tips for the Technically Gifted”.

This one looks at the value of automated software testing from the perspective of the business-side.  The reason for the focus is that most programmers I know and respect already understand the upside to automated testing and know way more than I do.  If this is you, feel free to stop reading.  I won’t be offended.

Business Thoughts On Automated Software Testing
 
Automated software testing is a large and relatively complex area that takes a while to understand.  But, let’s work with a simple definition:  It is the process of using computers (instead of humans) to run repeated tests to determine whether the software does what it is supposed to do.  It is important to note that most automated software testing still involves humans in the beginning (to design and develop the tests), but it’s the repeatability that makes it so powerful.  Once the tests are developed, the payback is continuous because the costs of running the tests are near zero.

In order to better illustrate my points, I’ll use Pyramid Digital Solutions (the first software company I started).  Pyramid ran successfully for 10+ years and was recently sold, but I like to use it as an example because I actually lived a lot of these lessons and I find it helpful to have a real-world example to talk about.

1. Build Better Software:  This one is obvious, but is at the core of the value so needs to be said.  By building a library of automated tests, you are generally going to ship better software that at least, at a minimum works when used in certain, predictable, preconceived ways (the use cases that have been accounted for in the tests).  This is a good thing.

2. Test Continuously:  As noted, once you have tests automated, there is very little cost to running the test.  As such, once you’ve made the investment in building automated test scripts, there is no good reason not to run them frequently (and lots of good reasons to do so).  In my prior startup, we eventually got to over 20,000+ test scripts that run for several hours.  We ran them every night.  Each night a process would fire off that would retrieve the latest source code the programmers had checked in, build our product (automated builds) and then run our test scripts.  Every morning, the results of the test scripts got emailed to management and the development team.  

 

3. Cheaper To Fix Bugs:  Most software has bugs.  From the business perspective, the questions are:  which bugs do you know about, when do you “find” them and how much does it cost to fix them?  As it turns out, when you find them and how much it costs to fix them are highly correlated.  Lets take an example.  From my prior (real-world) example, lets say a programmer inadvertently makes a code change and checks it in.  The code has a bug.  In the old way we used to operate, it often be days, weeks or months before that big got caught (based on what part of the product the code was in, whether it was caught internally, or made it out into the “wild” to be found by customers).  The more time that elapsed from the when the code actually changed, to when the bug was actually found, the more expensive the bug became to find and fix.  We’re talking major (orders of magnitude) increase in costs.  Now, in the new world (where we had automated tests running every night), this bug may be caught by the automated test scripts.  If so, the very next morning we would know there was a problem and we could go fix it. The reason it was so must cheaper to find and fix the bug was because the “surface area” of change was so small.  A limited number of things got changed in the prior 24 hours (since the last test), so the bug could more easily be discovered.  I cannot emphasize enough how much money you can save by catching bugs within hours (instead of days) of the bug being introduced.

4. Freedom To Change:  As software systems get bigger, it becomes harder and harder to make changes without breaking things.  Development teams do what they can to refactor the ugly bits of code as they can (and as time allows), but even then, a sufficiently large-scale codebase that has been around for a while will almost always have “corners” of it that nobody wants to touch (but are important).  The business risk to this situation is that you may find yourself in a situation where customers are asking for things or the market shifts in some way that causes the need for change (this should not come as a surprise).  If the programmers are fearful of changing core parts of the system for fear that they’ll break something, you’ve got a problem.  If you’ve got a large battery of automated test scripts, it frees the programmers to do lots of cool things.  They can refactor the code (the automated testing is a “safety net”), they can add/change features, etc. with a lot less loss of sleep.  What you will find, by investing in automated testing is that your organization actually moves faster than it did before.  You can respond to market change quicker, you roll out features quicker and you have a stronger  company.

5. Clients Are Happier:  At Pyramid, we had quarterly meetings with our clients (and an annual conference where a bunch of clients got together).  At each of these, one of the key metrics we shared was how large our automated tests were.  This gave clients some comfort.  This comfort translated into a higher likelihood that they would install newer versions of the software (when the became available).  Since we were in the high-end, enterprise software space, this was a big deal.  If we could get 20% more of our customers to move to Version 5 of our software (instead of staying stuck on Version 4), we had advantage.  Less support costs and higher retention rates.

I like to think of technology strategy in the form of technology debt (take short cuts now, but payback comes later – with interest).  Read “Short-Cuts Are Not Free” if you’re curious about this.  Similar to financial debt, this is often necessary (as is technology debt) but it has a cost.  The reverse of this is Technology Investment (in the classic sense).  This too has an interest rate – the rate you get “paid” (i.e. ROI) on that investment.  I think investment in automated testing is one of the best interest rates you can find.  The payback period is a little longer, but it is worth it.  If you have competition (which you likely will), you will find that having a strong investment in automated testing will give you advantage.  You’ll add features quicker, fix bugs cheaper and ship better software.

Of course, as is always the case, situations vary.  Pyramid was in a different market than my current startup HubSpot – but I’m still passionate about automated testing.  Will continue to share experiences as I progress.