I totally agree with you on the merits of automated testing. In the late 1990s I was on a project that used Vector Software's Computer Aided Software Test (CAST) tools and they were very helpful. A good CAST tool is hard to come by, though, I have seen some for Java that are expensive and do such a poor job of testing (only passing in the number "7" for an integer parameter, for example, instead of working the parameter through its ranges). So, caveat emptor when buying a CAST tool to help you build an automated testing framework. Some will help a lot and some are a joke, and you can't tell which is which by the price. You have to invest the time to actually test the tools on your real code.
Are you referring to nunit testing or something else in this article?
Referring to both unit testing (with things like Junit, Nunit and others) and "external" testing -- which is much harder. Basically, anything that improves the chances of finding bugs in an automated fashion.
Excellent article. Thank you for writing it.
Can you ballpark what the cost of developing the auto testing systems was on a percentage basis of overall development cost?
I would imagine it was in the low single digits?
I believe automated testing (particularly unit testing ) does deliver a competitive advantage.
The caveat with unit testing is that your product architecture must facilitate the economical development of unit tests (i.e. MVC  and perhaps SOA  as well? ), and you have the discipline to write tests!
It can be difficult to see the value of writing unit tests early in development, particularly when your product has few features and functional/user acceptance testing  can be done in minutes.
However, unit testing pays for itself tenfold as your product matures (and subsequently gets more complex).
1. Support for the built-in quality that repetitive automated testing is refreshing to see in the context of a startup – some things need to be done right even when in a lean release early crunch. My own experience has yielded great benefits from automated unit test. GUI testing has been much less successful.
2. I agree automated user interface functional testing is hard! Here lies a Web 2.0 warning: So far web apps have been simpler to test because the GUIs were simpler and typically followed simpler request/response dialogs. The sophisticated browser-based interfaces now possible present much more complexity in terms of test combinations.
3. Regarding "technology debt (take short cuts now, but payback comes later – with interest)". This is a great way to think about it -- sacrificing automated unit testing then would be like taking a loan from the loan shark in the worst neighborhood!
I posted a follow-up here:
You Can Pay Me Now, Or...
Your unsubscribe is broken, asshole. Can't you get this fixed?
b: Send an email to unsubscribe [at] onstartups.com and I'll make sure it gets taken care of.
Not sure if you're talking about subscribing to the comment notifications or the email list (am assuming comments, given your tone).
I couldn't agree more. In fact, my company builds a tool to assist in the automated scheduling, running and reporting of builds with tests. The only thing I would say is that you can (and should where possible) take point 2 even further: run the tests as frequently as possible. You should aim to get at least a fast subset of the tests to run in minutes rather than hours. Then run these tests on every new check in to the source repository. The advantage is as stated in point 3: but the feedback is even faster. You can track the exact change that caused the problem, and the developers find out ASAP. This also allows them to fix the problem before it affects other developers in their team. Those tests that can't be run quickly can be run on a separate, slower cycle (perhaps overnight as you did at Pyramid).
(Warning: shameless plug ahead) I wrote an article on the topic a little while back:
The Road to Build Enlightenment:
Interested readers should definitely also refer to Martin Fowler's article:
Article is good, but as some has pointed out its very obvious. But even then, its a nice reading. Even with all the benefits of automation testing, I have seen projects struggling with automation whenevr GUI is involved. Handling GUI from vendor supplied tool, when GUI is changing ever now and then is very difficult. Automation is certainly inevitable for API, Web Services or any product with good command line interface. For GUI, I guess 80/20 rule should be followed. For API testing This
page gives good information. But looks like site is still under construction.
First up - I represent a vendor - but not one the ones that instantly spring to mind when one is dismissing GUI testing tools.
Why - beacuse our GUI testing tools actually will work!!
Move the conntrols around the screen - no problems - we'll map them and then update the test script to match the new version of the appication.
Complex scripting language? no way .. far too labour intensive and one of the reasons why ther products are shelf ware.
What about creating automated tests when carrying out manual testing? - yep - only we do it.
Please contact me off-line at email@example.com. I hope I haven't offended or broken any of the rules on this forum, as I did state up front I was from a vendor.
Very good article, thank you! I'm trying to prove to our management the tests should move to the automation side, so your article will be a weapon, too. :)
Regarding (so much mentioned here in comments) GUI testing, there is a very good tool, both simply learned and powerful, we use it a lot: Macro Scheduler from http://mjtnet.com It can check a wide range of things, GUI included. It cannot go too much inside the components as WinRunner would, for example, but it covers pretty much. Not less important - it's light, doesn't record all the PC environment as WinRunner does when you record a new test, and it can also be compiled to light exe which can then be run at any other PC, at dev, QA, or customer's - Macro Scheduler doesn't have to be installed there (again, comparing to WinRunner, a huge advantage). Besides its own script possibilities, it includes VBScript, runs with your dlls, in short - can be used very widely.
And it's not a shameless plug in, because we're justtheir very happy customers. I've been working with this tool in several companies now, including my own, and I love it.