Reading notes on the book Lean Analytics: Use Data to Build a Better Startup Faster
PART ONE: Stop Lying to Yourself
Chapter 1: We’re All Liars
Small lies and delusions are essential to entrepreneur. They create your reality distortion field. They are a necessary part of being an entrepreneur. But if you start believing your own hype, you won’t survive.
You need to lie to yourself, but not to the point where you’re jeopardizing your business.
That’s where data comes in.
Guts matter; you’ve just got to test them. Instincts are experiments. Data is proof.
"If you can't measure it, you can't manage it." -- Peter Drucker
Case Study: Airbnb Photography
Airbnb started with a hypothesis: “Hosts with professional photography will get more business. And hosts will sign up for professional photography as a service.” (gut instincts)
Test the hypothesis: Concierge Minimum Viable Product (MVP)
The Minimum Viable Product is the smallest thing you can build that will create the value you’ve promised to your market.
If you’re considering building a ride-sharing service, for example, you can try to connect drivers and passengers the old-fashioned way: by hand.
Advantages:
- run things behind the scenes for the first few customers lets you check whether the need is real;
- helps you understand which things people really use and refine your process before writing a line of code or hiring a single employee
Test results: professionally photographed listings got two to three times more bookings than the market average. (validated hypothesis)
Experiement further, measure the results (by photo shoots, since it had already proven with its Concierge MVP that more professional photographs meant more bookings.), and adjust along the way:
- It watermarked photos to add authenticity.
- It got customer service to offer professional photography as a service when renters or potential renters called in.
- It increased the requirements on photo quality.
Summary:
Sometimes, growth comes from an aspect of your business you don’t expect. When you think you’ve found a worthwhile idea, decide how to test it quickly, with minimal investment. Define what success looks like beforehand, and know what you’re going to do if your hunch is right.
Chapter 2: How to Keep Score
In a startup, the purpose of analytics is to find your way to the right product and market before the money runs out.
What Makes a Good Mertic?
Rule of thumb:
- Comparative: being able to compare a metric across time periods, group of users, or competitors, etc. helps you understand which way things are moving.
- Understandable: easy for people to remember and discuss.
- A ratio or a rate:
- Easier to act on: distance vs. speed (distance per hour) = informational vs. actionable
- Inherently comparative: daily metric/monthly average metric -> possible trumpet or spikes or long-term trend
- Good for comparing factors that are somehow opposed, or for which there’s an inherent tension: e.g. distance/# of traffic tickets. The faster you drive, the more distance you cover—but the more tickets you get. (whether or not you to break the speed limit.)
- Changes the way you behave: (most important)
What will you do differently based on changes in the metric?- “Accounting” metrics, like daily sales revenue, show you how close you are to an ideal model and whether your actual results are converging on your business plan.
- “Experimental” metrics, like the results of a test, help you to optimize the product, pricing, or market.
if you want to change behavior, your metric must be tied to the behavioral change you want. If you measure something and it’s not attached to a goal, in turn changing your behavior, you’re wasting your time. (salesperson spend more time on persuading the customer for a good rating in the survey, which did not happen)
Metrics come in pairs.
- Conversion rate (the percentage of people who buy something) is tied to time-to-purchase (how long it takes someone to buy something). Together, they tell you a lot about your cash flow.
- Viral coefficient (the number of people a user successfully invites to your service) and viral cycle time (how long it takes them to invite others) drive your adoption rate.
How to chooose the right metrics?
Qualitative versus quantitative metrics
- Qualitative metrics are unstructured, anecdotal, revealing, and hard to aggregate;
- Quantitative metrics involve numbers and statistics, and provide hard numbers but less insight.
If quantitative data answers “what” and “how much,” qualitative data answers “why.” Quantitative data abhors emotion; qualitative data marinates in it.
Vanity versus actionable metrics
- Vanity metrics might make you feel good, but they don’t change how you act.
Actionable metrics change your behavior by helping you pick a course of action.
What will you do differently based on changes in the metric?e.g.
Vanity metrics:
“total signups”: The number can only increase over time. It tells us nothing about what those users are doing or whether they’re valuable to us.
“total active users”: a little better if you are defining the active user well. It still increase over time.
“# of hits”: problematic if your site has many objects. Count people instead.
“# of page views”: counts number of times someone requests a page, should count people instead.
“# of visits”: Is this one person who visits a hundred times, or are a hundred people visiting once?
“# of unique visitors”: only shows you how many people saw your home page. It tells you nothing about what they did, why they stuck around, or if they left.
“# of followers/friends/likes”: Counting followers and friends is nothing more than a popularity contest, unless you can get them to do something useful for you.
“#Time of site/number of pages”: poor substitute for actual engagement or activity unless your business is tied to this behavior. If customers spend a lot of time on your support or complaints pages, that’s probably a bad thing.
“Emails collected”: collected does not mean opened and acted on what’s inside them. Send test emails to registered subscribers instead to check.
“# of downloads”: though it affects ranking in app stores, it does not lead to read value. Measure activations, account creations, or something else.
Actionable metrics:
“% of active users”: This is a critical metric because it tells us about the level of engagement your users have with your product. It tells you whether the change you put into effect just now is working or not.
“# of users acquired over a specific time period”: this can help you compare different marketing approaches (fb campaign, reddit campaign, google adwords campaign in different weeks), and choose among them (actionable).
Exploratory versus reporting metrics
- Exploratory metrics are speculative and try to find unknown insights to give you the upper hand
Reporting metrics keep you abreast of normal, managerial, day-to-day operations.
Four kinds of information (Rumsfeld’s quadrants):
- Things we know we know: are FACTS that may be wrong and should be checked against data.
- Things we know we don’t know: are QUESTIONS we can answer by reporting, which we should baseline and automate. (Reporting: We know we don’t know the value of the metric, so we go find out.)
- Things we don’t know we know: are INTUITION, which we should quantify and teach to improve effectiveness and efficiency.
Things we don’t know we don’t know: are EXPLORATION, which is where unfair advantage and interesting epiphanies live. (Most relevent to startups in early stages: exploring to discover something new that will help you disrupt a market.)
Analytics has a role to play in all four of Runsfeld’s quadrants:
It can check our facts and assumptions—such as open rates or conversion rates—to be sure we’re not kidding ourselves, and check that our business plans are accurate.
It can test our intuitions, turning hypotheses into evidence.
It can provide the data for our spreadsheets, waterfall charts, and board meetings.
It can help us find the nugget of opportunity on which to build a business.
Case Study: Circle of Moms (discovered an “unknown unknown” that led to a big success)
Circle of Friends was a social graph application in the right place at the right time—with the wrong market.
By analyzing patterns of engagement and desirable behavior, then finding out what those users had in common, the company found the right market for its offering.
Once the company had found its target, it focused—all the way to changing its name. Pivot hard or go home, and be prepared to burn some bridges.
There’s a “critical mass” of engagement necessary for any community to take off. Mild success may not give you escape velocity. As a result, it’s better to have fervent engagement with a smaller, more easily addressable target market. Virality requires focus.
Leading versus lagging metrics
- Leading metrics give you a predictive understanding of the future;
- lagging metrics explain the past.
Leading metrics are better because you still have time to act on them—the horse hasn’t left the barn yet.
e.g.
Leading metrics:
- Current number of prospects in your sales funnel gives you a sense of how many new customers you’ll acquire in the future.
- Volume of customer complaints: the number of support calls that happen in a day, the number of customer complaints in a 90-day period, etc. These are leading indicators of churn: if complaints are increasing, it’s likely that more customers will stop using your product or service. You can also dig deeper into them to see the reasons behind and address those issues.
New qualified leads: they let you predict sales success ahead of time. For B2B, you also need a good understanding of conversion rate and sales-cycle length. Only then can you make a realistic estimate of how much new business you’ll book.
Lagging metrics:
Churn (which is the number of customers who leave in a given time period) gives you an indication that there’s a problem—but by the time you’re able to collect the data and identify the problem, it’s too late. We can indeed act on this metric (work to improve churn and then measure it again), but it’s akin to closing the barn door after the horses have left. New horses won’t leave, but you’ve already lost a few.
- Account cancellation or product returns: Both are important metrics—but they measure after the fact. They pinpoint problems, but only after it’s too late to avert the loss of a customer.
Quarterly new product bookings
In some cases, a lagging metric for one group within a company is a leading metric for another. The number of quarterlt bookings is a lagging metric for salespeople (the contracts are signed already), but for the finance department that’s focused on collecting payment, they’re a leading indicator of expected revenue (since the revenue hasn’t yet been realized).
Ultimately, you need to decide whether the thing you’re tracking helps you make better decisions sooner. As we’ve said, a real metric has to be actionable. Lagging and leading metrics can both be actionable, but leading indicators show you what will happen, reducing your cycle time and making you leaner.
Correlated versus causal metrics
- If two metrics change together, they’re correlated,
- If one metric causes another metric to change, they’re causal.
If you find a causal relationship between something you want (like revenue) and something you can control (like which ad you show), then you can change the future.
Usually, causations aren’t simple one-to-one relationships. Many factors conspire to cause something. You’ll get several independent metrics, each of which “explains” a portion of the behavior of the dependent metric. But even a degree of causality is valuable.
Correlation does not indicate causality. You prove causality by finding a correlation, then running an experiment in which you control the other variables and measure the difference. This is hard to do because no two users are identical; it’s often impossible to subject a statistically significant number of people to a properly controlled experiment in the real world.
For companies like Google and Microsoft who have a big enough sample of users, they can run a reliable test without controlling all the other variables, because eventually the impact of the other variables is relatively unimportant. But for the average startup, you’ll need to run simpler tests that experiment with only a few things, and then compare how that changed the business.
Correlations can help you predict what will happen. But finding the cause of something means you can change it. Sometimes, you may have to settle for the former—but you should always be trying to discover the latter.
Moving Targets
When picking a goal early on, we must know that we are chasing a moving target, because you really don’t know how to define success.
Sometimes there’s a huge gulf between what you assume and what users actually do. You might think that people will play your multiplayer game, only to discover that they’re using you as a photo upload service. Unlikely? That’s how Flickr got started.
Sometimes, however, the differences are subtler. You might assume your product has to be used daily to succeed, only to find out that’s not so.
In these situations, it’s reasonable to update your metrics and goals accordingly, provided that you’re able to prove the value created.
Case Study: HighScore House
HighScore House drew an early, audacious line in the sand—which it couldn’t hit.
The team experimented quickly to improve the number of active users but couldn’t move the needle enough.
They picked up the phone and spoke to customers, realizing that they were creating value for a segment of users with lower usage metrics.
Summary:
- Know your customers. All the numbers in the world can’t explain why something is happening. Interact with them directly.
- Make early assumptions and set targets for what you think success looks like, but don’t experiment yourself into oblivion. Use qualitative data to understand what value you’re creating and adjust only if the new line in the sand reflects how customers (in specific segments) are using your product.
Segments, Cohorts, A/B Testing and Multivariate Analysis
Testing is at the heart of Lean Analytics. Testing usually involves comparing two things against each other through segmentation, cohort analysis, or A/B testing.
Segmentation
- A segment is simply a group that shares some common characteristic.
On websites, you segment visitors according to a range of technical and demographic information, then compare one segment to another. Find the reasons behidn the unusual issues, and try to discover why.
Cohort Analysis
Cohort analysis compares similar groups over time, which is also called longitudinal studies. Users who join you in the first week will have a different experience from those who join later on.
Each group of users is a cohort—participants in an experiment across their lifecycle. You can compare cohorts against one another to see if, on the whole, key metrics are getting better over time.
If you simply look at the revenue across time, you can’t learn much, sicne you are mingling the purchases of a customer who’s been around for five months with those of a brand new one.
But if we break out the revenue by month in which that customer group started using the site, we can have a better understanding of how each customer group is contributing to the total revenue.
Or we can break out the revenue by the number of months they’ve used the system, and see how each cohort’s contribution goes across the time.
This kind of reporting allows you to see patterns clearly against the lifecycle of a customer, rather than slicing across all customers blindly without accounting for the natural cycle a customer undergoes. Cohort analysis can be done for revenue, churn, viral word of mouth, support costs, or any other metric you care about.
A/B Testing and Multivariate Analysis
Cohort experiments that compare groups are called longitudinal studies, since the data is collected along the natural lifespan of a customer group. By contrast, studies in which different groups of test subjects are given different experiences at the same time are called cross-sectional studies.
When we’re comparing one attribute of a subject’s experience, such as link color, and assuming everything else is equal, we’re doing A/B testing.
You can test everything about your product, but it’s best to focus on the critical steps and assumptions.
A/B tests seem relatively simple, but they have a problem. Unless you’re a huge web property—like Bing or Google—with enough traffic to run a test on a single factor like link color or page speed and get an answer quickly, you’ll have more things to test than you have traffic.
- Rather than running a series of separate tests one after the other—which will delay your learning cycle—you can analyze them all at once to see which correlates with a result using a technique called multivariate analysis. This relies on statistical analysis of the results to see which of many factors correlates strongly with an improvement in a key metric.
The Circle of Life for analytical startups
Chapter 3: Deciding What to Do with Your Life
We need to find the business that works (Lean Canvas) and also the business that we want to work on (Bud Caddell’s three criteria).
The Lean Canvas
The Lean Canvas is a one-page visual business plan that’s ongoing and actionable. As you can see in the below figure, it consists of nine boxes organized on a single sheet of paper, designed to walk you through the most important aspects of any business.
The Lean Canvas is fantastic at identifying the areas of biggest risk and enforcing intellectual honesty. When you’re trying to decide if you’ve got a real business opportunity, Ash says you should consider the following:
Problem: Have you identified real problems people know they have?
Customer segments: Do you know your target markets? Do you know how to target messages to them as distinct groups?
Unique value proposition: Have you found a clear, distinctive, memorable way to explain why you’re better or different?
Solution: Can you solve the problems in the right way?
Channels: How will you get your product or service to your customers, and their money back to you?
Revenue streams: Where will the money come from? Will it be one-time or recurring? The result of a direct transaction (e.g., buying a meal) or something indirect (magazine subscriptions)?
Cost structure: What are the direct, variable, and indirect costs you’ll have to pay for when you run the business?
Metrics: Do you know what numbers to track to understand if you’re making progress?
Unfair advantage: What is the “force multiplier” that will make your efforts have greater impact than your competitors?
What Should You Work On?
The Lean Canvas provides a formal framework to help you choose and steer your business. But there’s another, more human, side to all of this.
Do you want to do it?
If you’re going to survive as a founder, you have to find the intersection of demand (for your product), ability (for you to make it), and desire (for you to care about it).
Don’t start a business you’re going to hate. Life is too short, and your weariness will show.
Bud Caddell’s diagram
Ask yourself three questions when launching a new venture:
Can I do this thing I’m hoping to do, well? Never start a company on a level playing field—that’s where everyone else is standing. Don’t launch a new product or enter a new market unless your existing product and market affords you an unfair advantage.
whether you like doing this thing. You need to believe in what you’re doing so that you’ll keep at it and ride
be sure you can make money doing it. This is about the market’s need. This is by far the most important of the three; the other two are easy, because they’re up to you. But now you have to figure out if anyone will pay you for what you can and want to build.
Chapter 4: Data-Driven Versus Data-Informed
Rather than be a slave to the data, these critics say, we should use it as a tool. We should be data-informed, not data-driven.
Humans do inspiration; machines do validation.
Optimization is all about finding the lowest or highest values of a particular function, i.e. the local maximum.
Math is good at optimizing a known system; humans are good at finding a new one. Put another way, change favors local maxima; innovation favors global disruption.
Machine-only optimization suffers from similar limitations as evolution. If you’re optimizing for local maxima, you might be missing a bigger, more important opportunity. It’s your job to be the intelligent designer to data’s evolution.
Ultimately, quantitative data is great for testing hypotheses, but it’s lousy for generating new ones unless combined with human introspection.
How to Think Like a Data Scientist
10 common pitfalls that entrepreneurs should avoid as they dig into the data their startups capture.
- Assuming the data is clean. (issues behind NAs and values with high frequencies)
- Not normalizing. (comparing counts to totals)
- Excluding outliers. (for qualitative analysis)
- Including outliers. (for model building)
- Ignoring seasonality. (time of day, day of week, monthly changes in patterns)
- Ignoring size when reporting growth. (from 1 to 2 = double)
- Data vomit. (too much info displayed on the dashboard)
- Metrics that cry wolf. (Too sensitive thresholds)
- The “Not Collected Here” syndrome. (Focusing solely on the data sources that we have, ignoring external sources)
- Focusing on noise. (Fail to look at the bigger picture)
Lean Startup and Big Vision
How do you have a minimum viable product and a hugely compelling vision at the same time?
Big, hairy and audacious vision is important, but it has to reconcile with the step-by-step, always-questioning approach of Lean Startup. You need to think of Lean Startup as the process you use to move toward and achieve your vision.
We sometimes remind early-stage founders that, in many ways, they aren’t building a product. They’re building a tool to learn what product to build.
PART TWO: Finding the Right Metric for Right Now
Chapter 5: Analytics Frameworks
Dave McClure’s Pirate Metrics
McClure categorizes the metrics a startup needs to watch into acquisition, activation, retention, revenue, and referral—AARRR.
The table below shows our interpretation of his model, describing the five steps through which users, customers, or visitors must progress in order for your company to extract all the value from them. Value comes not only from a transaction (revenue) but also from their role as marketers (referral) and content creators (retention).
| Element | Function | Relevant metrics |
|---|---|---|
| Acquisition | Generate attention through a variety of means, both organic and inorganic | Traffic, mentions, cost per click, search results, cost of acquisition, open rate |
| Activation | Turn the resulting drive-by visitors into users who are somehow enrolled | enrollments, signups, completed onboarding process, used the service at least once, subscriptions |
| Retention | Convince users to come back repeatedly, exhibiting sticky behavior | engagement, time since last visit, daily and monthly active use, churns |
| Revenue | Business outcomes (which vary by your business model: purchases, ad clicks, content creation, subscriptions, etc.) | Customer lifetime value, conversion rate, shopping cart size, click-through revenue |
| Referral | Viral and word-of-mouth invitations to other potential users | Invites sent, viral coefficient, viral cycle time |
These five elements don’t necessarily follow a strict order—users may refer others before they spend money, for example, or may return several times before signing up—but the list is a good framework for thinking about how a business needs to grow.
Eric Ries’s Engines of growth
Eric Ries talks about three engines that drive the growth of a startup. Each of these has associated key performance indicators (KPIs).
Sticky Engine
The sticky engine focuses on getting users to return, and to keep using your product. (retention phase in McClure’s framework)
The fundamental KPI for stickiness is customer retention. Churn rates and usage frequency are other important metrics to track.
Long-term stickiness often comes from the value users create for themselves as they use the service (gmail, evernote, game account).
In terms of frequency, we also need to track metrics like time since last visit. If you have methods of driving return visits such as email notifications or updates, then email open rates and click-through rates matter, too.
Virality Engine
Virality is all about getting the word out.
The key metric for this engine is the viral coefficient—the number of new users that each user brings on. Because this is compounding (the users they bring, in turn, bring their own users), the metric measures how many users are brought in with each viral cycle. Growth comes from a viral coefficient of greater than one, but you also have to factor in churn and loss. The bigger the coefficient, the faster you grow.
Measuring viral coefficient isn’t enough. You also need to measure the actions that make up the cycle. e.g. Users sign up - connect to email - send invites to friends - friends sign up. Those distinct stages all contribute to virality, so measuring actions is how you tweak the viral engine—by changing the message, simplifying the signup process, and so on.
Paid Engine
The third engine of growth is payment. It’s usually premature to turn this engine on before you know that your product is sticky and viral.
e.g.
- focus on a beta group first (stickiness)
- invite friends to play (viriality)
- players make in-app purchase to enhance in-game experience (payment).
Getting paid is, in some ways, the ultimate metric for identifying a sustainable business model. If money earned from customers > money spent on acquring customers: sustainable
Revenue helps growth only when you funnel some of the money generated from revenue back into acquisition.
Then you have a machine that you can tune to grow the business over time.
Measure the balance between:
- Customer lifetime value (CLV)
- Customer acquisition cost (CAC)
but the equation for success isn’t that simple. You still need to worry about cash flow and growth rate, which are driven by how long it takes a customer to pay off. One way to measure this is time to customer breakeven—that is, how much time it will take to recoup the acquisition cost of a customer.
Ash Maurya’s Lean Canvas
Mentioned before in Chapter 3.
Sean Ellis’s Startup Growth Pyramid
His Startup Growth Pyramid, shown in the Figure below, focuses on what to do after you’ve achieved product/market fit.
The question this poses a of course, is how do you know if you’ve achieved product/market fit? Sean devised a simple survey that you can send customers (available at survey.io) to determine if you’re ready for accelerated growth. The most important question in the survey is “How would you feel if you could no longer use this product or service?” In Sean’s experience, if 40% of people (or more) say they’d be very disappointed to lose the service, you’ve found a fit, and now it’s time to scale.