The @Dharmesh Test: 16 Questions For Better SaaS Companies

About This Blog

This site is for  entrepreneurs.  A full RSS feed to the articles is available.  Please subscribe so we know you're out there.  If you need more convincing, learn more about the site.



And, you can find me on Google+

Connect on Twitter

Get Articles By Email

Your email:


Blog Navigator

Navigate By : 
[Article Index]

Questions about startups?

If you have questions about startups, you can find me and a bunch of other startup fanatics on the free Q&A website:

Subscribe to Updates


30,000+ subscribers can't all be wrong.  Subscribe to the RSS feed.

Follow me on LinkedIn


Current Articles | RSS Feed RSS Feed

The @Dharmesh Test: 16 Questions For Better SaaS Companies


The Dharmesh Test is a quick, simple set of yes/no questions to gauge the likelihood of long-term success of a SaaS (Software as a Service) company.  It is inspired by the highly popular “Joel Test” for measuring software development teams.  [Disclosure: I am an angel investor in Joel’s company Stack Overflow]

OnStartups YesNo

Now, if you know me, and you know Joel, you know that I'm no Joel.  He's smarter and is a better writer.  So, why did I have the audacity to write the "Dharmesh Test"?  Because I'm passionate about SaaS companies and have spent the better part of the last 6 years studying them and learning from folks that should have written this article.  And, I've taken what I've learned and applied it to my own company, HubSpot -- which is doing very well and scores a respectable 13/16 on the test (yes, I've got work to do).

This is alpha Version 0.80 of this test.  Will be iterating on it based on community feedback.

The Dharmesh Test: 16 Questions For Better SaaS Companies

1. Is there exactly one version of your software that services all users?

Or stated differently, have you resisted the temptation to have custom code for an individual customer or small group of customers?  Ideally, in steady state, the same master code-base would be servicing all customers.  Any customizations are done through application configuration or via APIs and plugins.  If you have different versions running temporarily to test new versions with a subset of customers, that’s fine.

2. Do you have a free trial?

Not only does a free trial increase your likelihood of getting more customers, it often improves the simplicity of the product.  Many companies can’t have free trials because the product is just too complicated and requires too much human assistance to really “try” it. That's a bad thing.  Freemium (where you have a free version of your product that's feature-limited, not time-limited) counts too. 

3. Do you track key metrics like CAC, LTV and cancellation rate?

CAC = Customer Acquisition Cost. This is the total sales expense and marketing expense (salaries, programs, etc.) over a specific period (lets say a month) divided by the number of customers acquired in that month.

Cancellation Rate / Churn Rate:  A percentage calculated as the number of customers that cancel in a month divided by the number of customer at the beginning of the month.

LTV =  Lifetime Value of a customer.  Based on how long the average customer is expected to stay (a function of cancellation rate) and the average revenue.

The reason these numbers are important is simple:  They are the key metrics that will help you drive a SaaS company.  It answers questions around how much capital you need, how much you should be investing in product, whether customers are generally happy (or not), etc.

Example:  KissMetrics gets this stuff (because that's what they do.  And, if you're not yet reading David Skok's For Entrepreneurs blog, you should be.  Brilliant material on this topic.

4. Are your prices published on your website?

It is decidedly sub-optimal if people have to get a custom quote or talk to a human just to know what your price is.  Increasingly, the market values simplicity and transparency. If you’re not providing easy access to information, potential customers will go elsewhere.

5. Can people start paying you (become a customer) without interacting with a human?

The way of the world is frictionless web sales.  There's nothing quite like the joy of having your revenues go up at 4:00 a.m. in the morning on a Saturday -- without anyone having to talk to anyone.

6. Do you make more money from users that are getting more value?

Is there some “variable” component to your pricing?  Do you charge based on number of users, storage, frequency of use or some other metric?  If you have the ability to make more revenue from customers over time (without having to upsell them), that’s high margin business and very desirable.  By the way, the ideal state is when there are "in product" upgrade paths.  Customer reaches a certain limit and "upgrades" (starts paying you more) without you having to do anything.

7. Does your application have a simple, clean API (Application Programming Interface)?

The web is not just about connecting static websites — it’s now about connecting applications.  Without an API, your application is an island.  It’s unlikely that you will be able to meet all the needs of your customers.  An API allows you to integrate with third-parties (and they to you) to deliver a more complete solution for the customer. These kinds of integrations are also great ways to spread the word about your product (the more people that integrate with you, the more reach you have).

8. Can customers export their data out of your system?

To get more people to use/buy your product, you need to reduce not just the barrier to entry, but also the barrier to exit.  The product should be what keeps customers to stay with you — not data lock-in.  If you try to use data lock-in, fewer people will pick you in the first place.

9. Can you deploy (and rollback) a new version of the software in one-step without downtime?

This is to measure the effectiveness of your technical operations and software release tooling. Companies that have easy ways to deploy/rollback software tend to have more confidence and release more often.  Companies that release often tend to do better than those that don't.

10. Is there an online community where users can interact with your team and with other users?

Ideally , everyone in your company should be interacting with customers.  Especially developers.  And, customers should be able to talk about you (even if it's negative) online and interact with other customers.  [Note: I'm not talking here about "automatic" online venues that form -- but something that you are actively fostering and encouraging]

11. Do new developers commit code on their first day at work?

This is partly to measure how difficult your development environment is to setup and what controls you have in place.  If it takes 4 days for a developer to get their development environment setup, something's wrong.  If you’re too scared to have a rookie commit code to production, because it might bring the entire system down, something's wrong (you haven’t built a strong enough safety net yet).  It shouldn’t be that easy to break your system.  [Nod to Eric Ries for this one].

Example:  Cheezburger Network doesn't show new employees where the bathroom is until they've committed code.

12. Do you release code to production at least once a month?

This is to check whether or not you’re agile.  The challenge these days is that everyone thinks they’re agile, so asking "are you agile" is ineffective.  The proof is in the pudding.  Real agile teams release code — frequently.  Everything else is secondary.

13. Do you maintain a centralized backlog of new product ideas, bugs and issues?

Ideally, not only would you have a central database of ideas, but you'd provide the ability for customers to submit ideas and vote on other people's ideas. 

14. Do you regularly run A/B (split) tests?

Great SaaS companies are great at testing and deciding with data.  You should be running regular tests (as in several times a month).  Example tests: Should we remove this feature (does anyone care)?  Should we increase our price to $75?  

Example: A company that does this really well is FreshBooks. If you run into Mike McDerment, ask him about how they built their product to facilitate testing.  (He probably won't tell you, but doesn't hurt to try).  I also think SEOmoz (hi Rand!) does a great job at this.

15. Do you invest in design and user experience?

Customer expectations (even for B2B companies) are high on the web.  Having crappy design might be OK if you happen to have a product that people can't help but use -- but most of us are not in such forgiving circumstances.  Great design not only helps you attract customers, it helps you keep them -- and get referrals.  Profitable design is when you spend effort in design that is centered around your business objectives. 

Examples:  I like Campaign Monitor a lot.  Heck, even their API documentation site is beautiful.  But there are other examples like the ever-popular 37signals

16. Does your company or product have a personality?

This is the squishiest of the questions -- but it's important.  Great software companies today have a personality and a voice.  They're distinctive and remarkable.  Whether it's through their blog, through the copywriting in the product (like error messages), or through in-person interactions, some companies just have personality.

Examples: Take a look at what Wufoo or ZenDesk does.  Quirky?  Yes.  Effective?  Absolutely.


So, how did you do?  Any questions you think are unimportant and should have been left out?  Any you'd add? 

Posted by Dharmesh Shah on Thu, Dec 16, 2010


This is an excellent benchmarking test for SaaS companies. Startups especially, should run this test once a month or at least once a qtr to see how they are faring against it. Thank you very much Dharmesh. Will share.

posted on Thursday, December 16, 2010 at 3:20 PM by soumya

Thanks Dharmesh! I created a spreadsheet with the 16 questions so that my team and I can track our progress toward 16 out of 16 (we have plenty of work to do). Please email me if you want me to send it to you to share. 
Thanks for your help and guidance. 

posted on Thursday, December 16, 2010 at 3:22 PM by Jonathan

Great post-- I think this can be applied more broadly, not just to SAAS, but any subscription service, or even repeat-business centric models. Sites like Gilt and Birchbox, for example, feel far from SAAS but can certainly be evaluated by this criteria and benefit from your advice. Thanks!

posted on Thursday, December 16, 2010 at 3:26 PM by Katrina

How about having  
"Do you have tried and true way to get customers aka a proven funnel" 
"Do you make it easy for your top customers to spread the word (and benefit from doing so)?" 
Hope this helps. 

posted on Thursday, December 16, 2010 at 3:31 PM by Jonathan Drake

Excellent article! We're tracking quite well on this, think we have 3 or 4 issues to work out.  
We're making an online video game that is free to play with microtransactions.

posted on Thursday, December 16, 2010 at 3:33 PM by Kris Morness

We launched our first SaaS ERP suite back in 2005 and now have nine additional suites and we currently are helping our first customer migrate off one our systems, they are staying on two of the others we have. (so I would argue the ease of leaving is also a critical point). 
We have over 22 million users of our systems worldwide (they are white-labeled so people don't know they are using our systems). However, in all our years of Cloud Computing and SaaS, it is never a simple 'yes' or 'no' given for larger systems like ERP and CRM systems, the migration of legacy data and applying business rules means it isn't just a simple turn it on (unless what you are offering is essentially an IT commodity). 
So the other side of the argument for the free trial is that if can be offered as a free trial, what value is it offering and if it is that simple to attract, it is that simple to leave. The evidence is quite the opposite on the worth of free trials or freeware versions when it comes to long-term sustainability and profitability (e.g. You are essentially marginalizing your own products. I would argue that having an online demo that both customers and competitors can try is better. This is how we have attracted competitors to become resellers. 
Other than those points, most of the points you have brought up are extremely useful, but not only for SaaS, but for all systems within a company. Good article. 

posted on Thursday, December 16, 2010 at 3:46 PM by James Wallis Martin

Excellent! Thank you. These are great points - and yes, we have a couple to address. I agree that there is something missing: the funnel. More importantly, the market that turns into the funnel.  
You can have all the items in your list checked, but you still need to know your market, verify it exists, and have a strategy for reaching them. Social media has made this easier than ever before. However, I have seen some wonderful software (and other products) built, only to find that no one will use them - or at least not pay for them. Everyone should do both primary and secondary research before going too far down the path. 
We've created an SaaS for entrepreneurs looking for feedback on their products. Currently we are focusing on the "mom" market since they have the buying power. If you have a product you want to get feedback about from moms, check us out 
Thanks again for the great Article!

posted on Thursday, December 16, 2010 at 4:00 PM by Shelley Straitiff

Super post - very practical stress tests! 
Perhaps I would add - do you have a motivating sales commission structure in place despite the longer term nature of cash flow from deals. 
Plus great metrics at Bessemer Venture Partners site

posted on Thursday, December 16, 2010 at 4:07 PM by Ian Smith

Great list. Check all 16 -

posted on Thursday, December 16, 2010 at 4:07 PM by Soham

Wonderful article Dharmesh 
Some quick suggestions, these may apply more to Saas model focused on selling to enterprises: 
Do you have anybody in the company who is doing anything other than coding, designing or selling ? 
If you have a salesteam is its compensation primarily (i.e. 80% or more ) based on commission as opposed to salary ?

posted on Thursday, December 16, 2010 at 4:16 PM by PJ Gupta

Great article Dharmesh, really timely for me. 
I've been thinking about using a freemium model, but now I'm thinking a free trial is the way to go. That way I don't need to decide on what's included in the free version vs. the premium version. I'm sure that's one reason why HubSpot has a free trial and not a free version... 
Thanks for posting!!!

posted on Thursday, December 16, 2010 at 4:40 PM by Alex Cook

One more. 
Are your (legal) Terms and Conditions easy for potential customers to understand . . .and at the same time protect your interests. 
OK, two. . . 
Do you have comprehensive Assignments for all Developments signed by everyone (yes everyone) in your company. 

posted on Thursday, December 16, 2010 at 4:45 PM by Michell L. Grenier, Esq.

Today 9/16 by Jan 1, 2011 13/16 the differenc ebeing e-commerce additions to the website and a user forum.

posted on Thursday, December 16, 2010 at 5:01 PM by Karl Treier

Hi Dharmesh! 
Great questions all. I'd also add some things like: 
Do you have a company culture where decisions and the reasons behind them are transparent? 
We're moving to a world where great people can't get motivated or believe in something they don't understand, making internal transparency critical to success. 
Does your software solve an problem people/companies can't solve themselves or does it just make a painful process easier? 
We've discovered that while both attract customers, the former is essential to building something that has a competitive advantage, and is a "company" not a "feature." 
Great post. 

posted on Thursday, December 16, 2010 at 6:43 PM by Rand Fishkin

I've been developing software for two decades. I've worked with a lot of other programmers in that time, mostly at startups.  
Developers who commit code on the first day are the worst kind of programmers. Their code is crap, it ends up having to be rewritten several times, it results in very difficult to diagnose and find bugs and comes from people who are too arrogant to think about how to solve a problem, and instead just start writing crap code. 
I know this represents the majority of so-called programmers these days. But it is not a positive trend. This is why unit tests are so popular- they know their code is crap and they need a safety net. 
Quality developers can think about a problem for a week and then write two weeks worth of the other developers code over the course of about 2 days because they've thought thru the solution. (And this ends up saving 2 months in redesign, rearchitecture and bugs over the next 9 months.)  
I know I'm not going to convince the arrogant developers, but I hope to reach someone whose willing to think about it and realize that you wouldn't hire an architect who finishes the engineering specs for your house on the first day.  
Writing crap code that has to be rewritten slows people and the company down. 

posted on Thursday, December 16, 2010 at 6:45 PM by Experienced Developer

Thanks Dharmesh, 
i am going to use this. Great compilation of good ideas.  

posted on Thursday, December 16, 2010 at 7:20 PM by Bharat Lohani

Great list, Darmesh. I am building what will be a SaaS software company, and consider this a great starting point. Agree with some of the additions, as well as some of the comments (about programmers writing crap code).  
Then again, Facebook is run by programmers that are encouraged to push new stuff out when they consider it ready' for first gen. 
The world (of software development) is changing and we all need to keep our eyes and ears open to be stay or become successful in the new environments. 

posted on Thursday, December 16, 2010 at 7:21 PM by joost allard

Again a great article with lots of insights. We just launched and we have not encountered many of these items yet - so I appreciate you mentioning them and helping us know what to look out for. Thank again.  

posted on Thursday, December 16, 2010 at 7:27 PM by Kimberly Deas

If you are going to charge customers through a subscription or pay-as-you-go model, the customers need to know what your competitors are charging for a non-SaaS model. Putting your competitors prices on your website along with your own, would be helpful.

posted on Thursday, December 16, 2010 at 7:37 PM by Hetal Shah

This is a good eye opener for companies who wanted to invest on SaaS. Nice.

posted on Thursday, December 16, 2010 at 8:29 PM by Bright

Great list -- and a checklist, too. Thank you Darmesh. 
We are not exactly a classic SaaS company, but we certainly feel that we are re-defining the term "software as a service"... so, do *all* 16 questions in your list apply to us? 
Most of them do, but question #2 is a bit of a struggle. How could one offer a free trial for a single-function app (by design)? 

posted on Thursday, December 16, 2010 at 9:04 PM by softwarecandy

These are some very good questions, unfortunately I was not able to answer the all positively, but this helps in creating some new goals to strive for. 

posted on Thursday, December 16, 2010 at 9:59 PM by Apartments

Great compilation for tests. One more important element, although it seems implicit, is Security. Majority enterprises have this question, what's your security policy?  
A recent study shows, security is rated highest in order to subscribe a SaaS app. Second is integration which is very well covered in #7.

posted on Thursday, December 16, 2010 at 10:38 PM by Shailesh Gogate

Great list Darmesh and I really enjoy the posts from this website. I got only 5/16, so we tons of improvement to do, but this list provides an excellent guiding light on what we need to accomplish. 
Again, thanks!

posted on Friday, December 17, 2010 at 7:29 AM by JJ McGrath

Most SAAS Companies I've seen releases updates every 4 to 6 months. Exporting should be the main feature.

posted on Friday, December 17, 2010 at 7:54 AM by Web hosting

Great article and very valid measuring compnents.

posted on Friday, December 17, 2010 at 8:12 AM by Doug Mielke

You did NOT write CAC instead of CoCA?!?! I'm forwarding this to Halligan.

posted on Friday, December 17, 2010 at 11:58 AM by Tom Rose

This is a home-frickin' run. I'm modeling my company around this AND forwarding it to my boss.

posted on Friday, December 17, 2010 at 12:01 PM by Tom Rose

I'll go along with everything except 11. SAAS development is a lot different from web site development. You are developing applications rather than content. The idea that you would release code to production without running a full test suite is unthinkable. To force a new developer to master perhaps thousands of lines of dependent code and make a release decision in one day is also unthinkable, let alone foolhardy. There is no daily build that is fault-tolerant enough to consistently survive check-ins for the sake of checking in, rather than the release of working and "tested" code.

posted on Saturday, December 18, 2010 at 5:16 PM by Dave Barber

Hi Dharmesh, 
Great list. We scored 12/16 for KooKoo ( need to improve :)

posted on Sunday, December 19, 2010 at 11:55 PM by Chaitanya

Hey Dharmesh, awesome, as always, very useful...

posted on Monday, December 20, 2010 at 3:46 AM by Rosendal

Dharmesh, this is a great list although I might qualify the answer to no. 4, Are your prices published on your website?  
Although, transparency is fundamentally good, it is also a matter of the price point for your services. The more expensive the offering, the more difficult it is to assume that everyone understands you are selling “apples” when others may be selling “oranges.” Even if you can assume that, a prospect may be turned away by your published price not knowing how their specific budget needs can be accommodated. I’m not talking about discounting, but invoicing across different budget years or multiple year subscriptions.  
We are still a long way from eliminating the need for sales and marketing reps to interface with prospects.  

posted on Monday, December 20, 2010 at 5:45 PM by Larry H

Free trials with instant login or usability would be great. No one likes to wait 24 hours or even 48 hours for the trial link to make it's way to your inbox.

posted on Tuesday, December 21, 2010 at 2:50 AM by Web hosting

Great tips. We'll use this on our pending cloud offering.  
However, I disagree with one of the points: having a free trial. I agree that it's important to have an easy path to adoption and upgrade. But often when a firm offers a free trial it gets a lot of people who value the service offering exactly that much. And free-booters tend to be the most difficult. To guard against this, money has to change hands. A company can achieve the best of both worlds by offering an ultra-cheap trial cost. AWeber and several other SaaS providers have successfully adopted this model. 
Enjoyed your column. Thank you! 
the Business Doctor 
<a href>Grow Your Business

posted on Tuesday, December 21, 2010 at 9:44 AM by Chris Blanton

Great questions Dharmesh, thanks for sharing!  
To founders & CEOs SaaS companies: Guys, if you want to make your life easier when tracking revenue, churn, customer lifetime value, cohorts, etc - check out Sunnytrail

posted on Wednesday, December 22, 2010 at 5:21 AM by Octav Druta

These are all good points which are 'common sense' alas, common sense ain't that common. I'm trialling a rather expensive bit of kit, where the providers do not get full marks on any of the points raised. The price is published, for example, but there are a bucketload of extras, that increase the price by 50%.

posted on Thursday, December 23, 2010 at 11:52 PM by lasting power of attorney

Great post. Two comments... 
As a bootstrap enterprise SaaS startup I see it as impossible to provide a free trial. We're looking at providing other benefits during a trial period like some free training hours and free data import but to offer the system for free at this early stage would not be sustainable.  
The other issue is with putting our prices online. I think we can maximize our profits by pricing on a company by company basis but maybe you're right and we should be more open. 

posted on Friday, December 24, 2010 at 11:52 AM by Big Tuna

Hi Big Tuna, 
A few red flags with what you just said. People today really need a way to experience your service before they pay into it. I'd recommend the book "Four Steps to the Epiphany" by Steven Gary Blank. It talks about customer development. In your case, are you possibly skipping the whole phase of "1. Are customers even interested in my product or service, 2. Would customers pay for my service?, 3. Can I make it profitable? 
As soon as you get into pricing based on the affluence of your potential customer, you are playing with fire. People don't want to mess around with bartering these days.

posted on Friday, December 24, 2010 at 11:58 AM by Kris Morness

Comments have been closed for this article.