OnStartups

We Like Humans, But We LOVE Code (Machine Learning Case Study)

Posted by Dharmesh Shah on April 16, 2015 in guest development 6 Comments

The following is a guest post by Jonah Lopin (@jonahlopin), founder at Crayon.  Jonah is a HubSpot alumnus, and I'm an angel investor in the company.  

Here at Crayon, we love humans. 

In fact, all our moms and many of our best friends are humans! 

But when it comes to solving problems, we prefer code.

Humans are awesome at certain things, like closing enterprise sales deals, delivering rousing speeches, and comforting small children. For these tasks and lots of others, humans are far superior to code.

But in a business context, humans have some serious drawbacks:

1. They’re hard to recruit (Crayon is hiring, by the way)

2. They’re expensive (compared to servers)

3. They sometimes get sick and can’t come to work (☺)

For a product-driven software company, there’s something else to consider. It’s subtle, but important.

Software is better than humans at providing an elegant solution to complex problems at scale.

Tomasz from Redpoint put together some fascinating data back in 2012 that showed billion dollar public SaaS companies have revenue per employee around 200k/year. Jeremiah Owyang puts revenue per employee at Google & Facebook around $1m per employee, about 5x higher! Clearly, Facebook & Google solve more problems with code than a typical B2B software company.

The fact that Google & Facebook solve their primary problems with code rather than people doesn’t just impact their metrics and margins, it shapes the elegance and efficiency of the solutions they provide. The subtle advantage of solving problems with code rather than humans is that it tends to make your core product better over time.

Crayon is small, but we dream big. We solve problems with code, not humans, because we want to serve millions of users as elegantly as possible. It’s just down-right hard to do that with too many humans in the mix, especially if you want to scale quickly. (And especially if you’re not Uber.)

The rest of this post is about a big problem we had and how we solved it with code.

The Problem: Some Web Pages Don’t Look So Hot

Crayon is a visual inspiration platform. We help marketers get great ideas so they can do better marketing.

We’re programmatically adding about half a million pages to our system every day. As I write this article, we’ve got about 13 million designs in the system, and we’ll have close to 100 million by the end of the year.

We let users vote designs up and down, so most categories on Crayon like Startup Home Pages, B2B Pricing Pages and Landing Pages have always looked pretty good.

But we still had a problem in deeper categories: folks would be happily browsing, only to have the smooth inspirational vibe unceremoniously broken by a crappy looking page. How could we get the “best” designs to the top of the result set and the “least inspiring” designs to the bottom?

We Challenged Ourselves to Solve it With Code

Can code separate “inspiring” marketing designs from “crappy” ones?

Many said it couldn’t be done… but if they were right, would I be writing this article?

Here’s How We Solved It

Step 1: Pick Training Sets

We picked a set of 200 “inspiring” marketing designs and a set of 200 “uninteresting” marketing designs.

Yes, this is a bit of a fuzzy thing to do because it’s based on human judgment. But after running the services team at HubSpot for 5.5 years, and working closely with more than 8,000 professional marketers, I felt qualified to judge which marketing designs were likely to be interesting and instructive to other marketers. So sue me!

Step 2: Make Some Guesses About Why The Good Ones Are Good

This part is like picking a basketball team without being able to watch the candidates play basketball. What characteristics of the players would you look at? For instance, you might pick taller players or players wearing high-tops.

In the style of Jeff Foxworthy’s comedy routine You Might Be a Redneck:

If your Html is all based on tables… you might not have a great marketing design (tweet)

If you’ve got lots of inline styles… you might not have a great marketing design (tweet)

If you don’t use media queries in your CSS… you might not have a great marketing design (tweet)

You get the idea.

Step 3. Test Your Guesses

We wrote some code to test each “guess” from step 2 against each design in the training sets from step 1. We were hoping to find things that were “true” for the “inspiring” pages, and “untrue” for the “uninteresting pages”.

We looked at 45 discrete “guesses”, and found 25 of them were predictive of “inspiring” marketing designs. Success!

Some of the best factors were things like:

  • Setting the viewport meta tag
  • Including Facebook Open Graph tags
  • Using the Bootstrap framework
  • Specifying Apple touch icons

Note that these factors don’t directly predict which pages are “inspiring”. Rather, these factors indicate that someone clueful created the page, which is directly predictive of a page being “inspiring”.

Some of the things we thought might work, but weren’t predictive at all include:

  • Whether a page uses JQuery doesn’t matter… it’s just too ubiquitous
  • The total text on the page doesn't matter.  It turns out there are equally as many crappy short pages as there are long pages (tweet)

The final step was some mathematical mojo based on how strong the signals were in step 2 to come up with an overall “inspiringness score” for each page.

The Results

Pages like the chatterbox.co homepage did very well:

interesting-example

While our friends at biomarkerstrategies.com didn’t fare as well:

uninteresting-example

 

Overall, here’s what we got:

humans-vs-code-chart-chart

How awesome is that?

The top 10% of Crayon search results just went from 50% “inspiring” to a whopping 93%!

We have 13 million designs in the system today, and reviewing those manually would take a 10 person team about 5 years. Ouch! But we’ll have 100 million designs by the end of the year, and that’s just the beginning. If we didn’t solve this problem with code, we’d be in serious trouble. And no humans were involved… except for the one human writing this article.

Your thoughts?

Does your business have a tough problem you plan to solve with code rather than humans?

Have you used machine learning to deliver elegant solutions to customers at scale?

Please continue the converstaion in the comments.

Go solve something with code! (tweet)

Read More

5 Mistakes Every Startup Founder SHOULD Make

Posted by Dharmesh Shah on June 2, 2014 in guest strategy 11 Comments

“It’s fine to celebrate success but it is more important to heed the lessons of failure” - Bill Gates

Much digital ink has been spilled trying to caution startup entrepreneurs against making mistakes. Type "mistakes entrepreneurs make" into Google and you'll find thousands of articles, sternly forewarning against the most prevalent pitfalls and errors that stand between you and your startup's success.

But wait a second. How many times have you heard "We learn from our mistakes"? As an entrepreneur and investor, I've definitely made some doosies - and if my experience is any indication, not only do we learn from our mistakes, but we learn much more from our mistakes then from our successes.wrong-decision

Then why try so desperately to avoid them? Call me contrarian, but I'd argue that well-positioned mistakes can be quite worthwhile, and should even be encouraged.

So here's an initial list of mistakes that you SHOULD make as a startup founder. Mistakes that are bound to happen anyway. Mistakes you will learn so much from that you'd best make them as early in your entrepreneurial journey as possible. I've also included some recommendations about what to do after you make each mistake.

1. Get Screwed

It's inevitable. Someone - your partner, co-founder, employee, investor, or any other character in your unfolding plot - will mess you over. Someone will break your trust, violate a verbal or even written agreement, cut your compensation, or try to steal your equity or destroy your whole company (or all of the above, if you're me). Someone will do something stupid to scuttle your grand plan.

Accept the inevitable, steady yourself now for the oncoming blow, and just hope it doesn't hurt too much or cause too much damage. A startup is usually an odd mix of people (idea people, tech people, investors and others) thrown together by fate and circumstance in pursuit of a distant, moving target. As clearly as you think you see that target and the path and steps toward it, somebody else will see that path differently or may have a different set of scruples or incentives. That will create friction, and depending on the power balance between everyone involved, can result in someone - maybe you - getting screwed.

Post-Mistake Action: You're thinking, "Why is getting screwed my 'mistake?' I didn't do anything wrong." Look in the mirror. Upon reflection, you'll likely find that what enabled your misfortune was something you did or didn't do. The screwer-screwee relationship requires at least two people, and there are two sides to every story. Even if you clearly weren't "at fault" - you encountered a terrible, crooked person who did you in - you still need to ask yourself how you allowed yourself to do business with that person. Was the person's action foreseeable? Did you do your due diligence on your partner/employee/investor? What did you do or not do that exposed you? As George W. Bush famously said: "Fool me once, shame on you. Fool me twice… shame on…. we can't get fooled again!" (cue The Who). Having been burned once, you'll be much more careful in the future about compromising your principles or allowing yourself to do business with people who may take advantage of you later on.

2. Seek Revenge

This is an adjunct to the above mistake. Once bitten, your natural impulse may be to bite back. You've lost something - tangible, emotional, some future upside or all of the above - and you want to deny the perpetrator those same things or at least the satisfaction of having caused you that loss. You've been wronged, and your reflexive urge is to right the wrong by wronging back.

Go ahead, try it once. I predict that not only won't you be successful, but most likely nothing will happen at all, or worse, it will bounce back at you. If you got screwed, that probably indicates you don't have the leverage, power or ability to exact meaningful revenge anyway. You'll just feel immature, cheap and dirty and the lingering recollection of that bad feeling probably will be enough to prevent you from playing the revenge card again.

Post-Mistake Action: Look forward, not back. The objective of the startup game is to win, and two losses don't equal a win. Rather than dwell on the past, "Revenge it Forward." Your best revenge is going to be your own success. Use what happened to you as an additional driving force, a motivator to prove to yourself, the person who messed you over, and the world, that not only do you deserve better, but that you can achieve better.

3. Tell People Your Venture is in "Stealth Mode"

It's natural to want to keep your cards close to your vest. Perhaps you're afraid someone will steal your idea, or you lack confidence that you've developed it well enough to convincingly describe it to others. The tech industry has even provided you the gift of a cool-sounding cover: "Stealth Mode," which makes you sound more like a covert spy shrouded in secrecy than an unsure rookie plagued by insecurity.

Saying you're in "Stealth Mode" is almost certainly a mistake, for many reasons. First of all it can easily be interpreted as either pompousness or insecurity, which is bad for your credibility. You're also signaling that you don't trust that person, creating a negative feeling that will likely persist even after you're able to elaborate later on. Most importantly, though, you are missing out on potentially invaluable opportunities to use your own network to shape, develop and advance your product or venture. Every person you know, meet or speak with can be a key to your venture's ultimate success.

Post-Mistake Action: Switch to "Get Out There" Mode. You never know who may be a potential customer or investor for your awesome new product, or more likely, who may know someone who could be a customer or investor. There are ways to plant the seeds of curiosity, to scout out a territory, to indicate what market you are targeting, without giving away the store. As Steve Blank advocates, it's critical for the founder to "get out of the building" (physically or virtually) as early as possible, and to start getting feedback from live people very early on and keep iterating based on feedback. By keeping everything a closely-guarded state secret, you are squandering opportunities to connect with people who might ultimately help you achieve Product/Market Fit, partner with you or invest in your venture. Come up with a "teaser" line that tells people enough about what you're working on and who you're targeting to pique their interest, generate confirmation of market problems and pain points, and generate future interest or relevant contacts.

When I founded EverMinder I was happy to discuss it with just about anyone long before it was fully developed. I got great feedback that led me to our first three angel investors, all via referral, all before we launched.

4. Believe that "If You Build It They Will Come"

The problem with the movie "Field of Dreams" is that it planted the phrase "If you build it, they will come" firmly into the common parlance and specifically, into the heads of countless, impressionable startup founders. The popularity of the phrase (and its confirmation in the movie, when Costner builds "it" and "they" magically come) leads some founders to believe, and predict to investors, that they, too, need only to build their amazing new thingy, and the users will come running until the rest looks like a hockey stick.

The problem is that as opposed to ghostly baseball players in Hollywood movies, I can assure you that if you just build "it", "they" will almost certainly not come. In startup theory the "coming" of "they" is called "Market Pull" which almost never happens by itself, even among early adopters. Market Pull needs to follow an intense and iterative period of product design, customer development, Product/Market Fit and hands-on "Technology Push" into the target market, which only if successful begets the glorious Market Pull. You'll have to work hard to make the market notice and care, and probably personally engage your early users individually, and that's fine. If you've hit Product/Market Fit squarely, "they" will eventually come, but only after you've very actively recruited and engaged your early adopters.

Post-Mistake Action: Stop believing stupid movies. If you've quoted "if you build it they will come" then you don't understand enough about Technology Push and Market Pull, customer development, and developing your Minimum Desirable Product. Instead of watching movies, read. A lot. You need to learn about these concepts by immersing yourself in the writings of Steve Blank, Sean Ellis, Andrew Chen, this very blog, and the other great proponents of effective product and customer development, and get up to speed fast.

(Ok, if you're still looking to be inspired by a movie, then watch The Pursuit of Happyness. Listening to voices in a cornfield will get you nowhere. Perseverance in the face of adversity, hustle, understanding and caring passionately about your product and customers, and relentlessly pursuing your goals will give you a shot to win.)

5. "My Favorite Mistake"

This mistake is probably my favorite because, like the Sheryl Crow song, it's complicated. It's the mistake I continue to commit most frequently.

My favorite founder mistake is not appropriately balancing confidence and humility. There's a yin/yang relationship between the two and as you pilot your rocketship forward, you will occasionally find that you’ve leaned too hard to one side or the other.

As a startup founder you need to have a healthy dose of self-confidence. Ok, maybe an unhealthy dose. An overdose. You need to passionately believe that your solution is The Next Big Thing. But overconfidence can be extremely dangerous, for many reasons. It can be misinterpreted by others as arrogance, which can cause damaging interpersonal consequences. If overconfidence morphs into false confidence, It can cloud your vision or your analysis. A great founder must have just as healthy a dose of humility, an understanding of his or her relatively small place in the world. But being too humble can hold you and your venture back….

A great Talmudic sage once wrote "Every person should have two pockets, each with a reminder note that he should refer to in the appropriate circumstance. In one pocket should be a note with the phrase: 'The world was created just for me.' In the other pocket should be a note that reads: 'I am like the duct of the earth.' "

(Five bucks says that's the first time Sheryl Crow and a Talmudic sage were mentioned together.)

Post-Mistake Action: Unfortunately I don't have a clear "post-mistake" recipe for this one. Managing your self-confidence and humility is a push-and-pull balance exercise that you have to keep performing, every day. It's up to you to gauge each circumstances and determine which reminder note applies.

Now that I'm an early-stage investor, I meet with entrepreneurs eager to convince me that they (and their ventures) are awesome. That's great. But I don't want entrepreneurs to sugar-coat their backgrounds. I expect and want them to tell me about their mistakes and failures. I want to hear what they've learned (and also that they've made those mistakes already on someone else's nickel, not mine). A good entrepreneur wants to talk about their mistakes as well as their successes, and a good investor wants to hear about those mistake and lessons without penalizing the pitch.

It's cliché, but nobody's perfect. You're not perfect. Mistakes will happen, and you will make your share. Expect them, embrace them, and analyze them, as those mistakes and the lessons learned will become important, lasting building blocks in your personal development and the development of your company.

So get out there, make some mistakes, learn from them…. And win.

This was a guest post by Ben Wiener (@BeninJLM). Ben is a startup founder and is Managing Partner of Jumpspeed Ventures, a Jerusalem-based micro-fund that invests in early-stage startups.

Read More