Lean Analytics

Alistair Croll & Benjamin Yoskovitz

2013

Use Data to Build a Better Startup Faster

PART I - Stop Lying to Yourself


Chapter 1 - We're All Liars

"Without productivity objectives, a business does not have a direction. Without productivity measurements, it does not have control."

Airbnb Photography - A Concierge MVP example


Chapter 2 - How to Keep Score

What makes a good metric?

  • comparative
  • understandable
  • a ratio or a rate
  • easier to act on
  • inherently comparative
  • good for optimising on conflicting factors
  • changes the way you behave - "If you want to change behavior, your metric must be tied to the behavioral change you want."

Metrics dimensions

  • qualitative vs. quantitative
  • vanity vs. actionable
  • exploratory (unknown unknows) vs. reporting (known unknows)
  • leading vs. lagging
  • correlated vs. causal

Vanity Metrics to watch out for

  • number of hits
  • number of page views
  • number of visits
  • number of unique visitors
  • number of followers/friends/likes
  • time on site / number of pages
  • emails collected
  • number of downloads


Chapter 3 - Deciding What to Do with Your Life

"Markets that don't exist don't care how smart you are." - Marc Andreesen

Lean Canvas

What SHOULD I work on?

  • am I good at it?
  • do I enjoy iy?
  • can I get paid doing it?


Chapter 4 - Data-Driven vs. Data-Informed

"Humans do inspiration, machines do validation."

Data Science pitfalls (Monica Rogati)

  • assuming the data is clean
  • not normalising
  • excluding outliers
  • including outliers
  • ignoring seasonality
  • ignoring size when reporting growth
  • data vomit
  • metrics that cry wolf
  • excluding third-party data sources
  • focusing on noise

"Early-stage founders aren't building a product: they are building a tool to learn what product to build."



PART II - Finding the Right Metric for Right Now


Chapter 5 - Analytics Frameworks

Dave McClure's Pirate Metrics - AARRR

  • Acquisition - traffic, mentions, cost per click, search results, cost of acquisition, open rates
  • Activation - enrollments, signups, completed onboarding process, used the service at least once, subscriptions
  • Retention - engagement, time since last visit, daily and monthly active use, churns
  • Revenue - customer lifetime value, conversion rate, shopping cart size, click-through revenue
  • Referral - invites sent, viral coefficient, viral cycle time

Eric Ries's Engines of Growth

  • Sticky Engine
  • Virality Engine
  • Paid Engine - Customer Lifetime Value (CLV) vs. Customer Acquisition Cost (CAC)

Ash Maurya's Lean Canvas

Sean Ellis's Startup Growth Pyramid

  1. Product/Market Fit - decide what you sell to whom, then prove it
  2. Stack the Odds - find a defensible unfair advantage and tweak it
  3. Scale Growth - step on the gas in new markets, products, and channels

survey.io "How would you feel if you could no longer use this product or service?" 40%+ = market fit

The Long Funnel

The Lean Analytics Stages and Gates

  1. Empathy - I've found a real, poorly met need a reachable market faces
  2. Stickiness - I've figured out how to solve the problem in a way they will accept and pay for
  3. Virality - I've built the right product/features/functionality that keeps users around
  4. Revenue - The users and features fuel growth organically and artificially
  5. Scale - I've found a sustainable, scalable business with the right margins in a healthy ecosystem | I can achieve a successful exit for the right terms



Chapter 6 - The Discipline of One Metric that Matters (OMTM)

Four Reasons to Use the One Metric that Matters

  • it answers the most important question you have
  • it forces you to draw a line in the sand
  • it focuses the entire company
  • it inspires a culture of experimentation

Drawing Lines in the Sand

The Squeeze Toy - Optimizing your OMTM not only squeezes that metric so you get the most out of it, but it also reveals the next place you need to focus your efforts.


Chapter 7 - What Business are You in?

"Selling more stuff to more people more often for more money more efficiently." Sergio Zyman, Coca-Cola CMO

About those People - not all customers are good: optimise for good customers

The Business Model Flipbook

  1. Acquisition Channel
  2. Selling Tactic
  3. Revenue Source
  4. Product Type
  5. Delivery Model


Chapter 8 - Model One: E-commerce

...


Chapter 9 - Model Two: Software as a Service (SaaS)

  • Attention
  • Enrollment
  • Stickiness
  • Conversion
  • Revenue per Customer
  • Customer Acquisition Cost (CAC)
  • Virality
  • Upselling
  • Uptime and Reliability
  • Churn
  • Customer Lifetime Value (CLV)

Measuring Engagement

Churn (no login in 90 days or less) - using churn to compute CLV

Key Takeaways

  • While freemium gets a lot of visibility, it's actually a sales tactic, and one you need to use carefully.
  • In SaaS, churn is everything. If you can build a group of loyal users faster than they erode, you'll thrive.
  • You need to measure user engagement long before the users become customers, and measure customer activity long before they vanish, to stay ahead of the game.
  • Many people equate SaaS models with subscription, but you can monetize on-demand software in many other ways, sometimes to great effect.


Chapter 10 - Model Three: Free Mobile App

...


Chapter 11 - Model Four: Media Site

...


Chapter 12 - Model Five: User-Generated Content

...


Chapter 13 - Model Six: Two-Sided Marketplaces

...


Chapter 14 - Which Stage are You at?

  1. Empathy
  2. Stickiness
  3. Virality
  4. Revenue
  5. Growth


Chapter 15 - Stage One: Empathy

Summary

  • Your goal is to identify a need you can solve in a way people will pay money for at scale. Analytics is how you measure your way from your initial idea to the realisation of that goal.
  • Early on, you conduct qualitative, exploratory, open-ended discussions to discover the unknown opportunities.
  • Later, your discussions become more quantitative and more convergent, as you try to find the right solution for a problem.
  • You can use tools to get answers at scale and build up an audience as you figure out what product to build.

Qualitative feedback, interviews (15 people at least)

Finding a Problem to Fix (or, How to Validate a Problem)

  • The problem is painful enough
  • Enough people care - addressable market that are homogeneous within and heterogeneous between.
  • They're already trying to solve it - The current solution, whatever it is (even manual), will be your biggest competitor at first, befause it's the path of least resistance for people.

Signs You've Found a Problem Worth Tackling

  • they want to pay you right away
  • they're actively trying to (or have tried to) solve the problem in question
  • they talk a lot and ask a lot of questions demonstrating a passion for the problem
  • they lean forward and are animated (positive body language)

Negative Patterns

  • they're distracted
  • they talk a lot, but it's not about the problem or the issue at hand (they're rambling)
  • their shoulders are slumped or they're slouching in their chairs (negative body language)

Conduct PROBLEM interviews - focus on the problem alone

Running Lean and How to Conduct a Good Interview

  • Aim for face-to-face interviews
  • Pick a neutral location
  • Avoid recording interviews
  • Make sure you have a script
  • Set the stage for how the interview works
  • Test the customer segment by collecting demographics
  • Set the problem context by telling a story
  • Test the problem by getting the subject to rank the problems
  • Test the solution
  • Ask for something now that you're done - solution interview, referral for other interview

How to Avoid Leading the Witness

  • Don't Tip your Hand - no biased wording, open-ended questions
  • Make the Questions Real - ask for commitment/payment/referrals/introductions
  • Keep Digging - ask 'why?' several times, use uncomfortable silence

Look for Other Cues - non-verbal signs, use the 'Columbo' question

How Do I Know if the Problem is Really Painful Enough?

  • Did the interviewee successfully rank the problems you presented? (0-10)
  • Is the interviewee actively trying to solve the problems, or has he done in the past? (0-10)
  • Was the interviewee engaged and focused throughout the interview? (0-8)
  • Did the interviewee agree to a follow-up meeting/interview where you would present your solution? (0-8)
  • Did the interviewee offer to refer you to others for interviews? (0-4)
  • Did the interviewee offer to pay you immediately for the solution? (0-3)

32+/43 is good

Startup regularly underestimate the power of "good enough" solutions - mismatched socks are a universal problem nobody's getting rich fixing

Surveying

Building the MVP - keep gathering feedback and acquiring early adopters -> critical mass of testers and early adopter

Measuring the MVP - user acquisition is irrelevant, focus on making your users really happy

  • Are people using the product?
  • How are they using the product?
  • Are they using all of the product or only pieces of it?
  • Is their usage and behaviour as expected or different?

Be Prepared to Kill Features

Should I Move to the Next Stage?

  • Have I conducted enough quality customer interviews to feel confident that I've found a problem worth solving?
  • Do I understand my customers well enough?
  • Do I believe my solution will meet the needs of customers?


Chapter 16 - Stage Two: Stickiness

Summary

  • Your goal is to prove that you've solved a problem in a way that keeps people coming back.
  • The key at this stage is engagement, which is measured by the time spent interacting with you, the rate at which people return, and so on. You might track revenue or virality, but they aren't your focus yet.
  • Even though you're building the minimal product, your vision should still be enough to inspire customers, employees, and investors - and there has to be a credible way to get from the current proof to the future vision.
  • Don't step on the gas until you've proven that people will do what you want reliably. Otherwise, you're spending money and time attracting users who will leave immediately.
  • Rely on cohort analysis to measure the impact of your continuous improvements as you optimise the stickiness of your product.

Segment users & metrics by cohorts.

The Goal is Retention - Questions to Ask Yourself Before Building a Feature

  • Why will it make it better?
  • Can you measure the effect of this feature?
  • How long will the feature take to build?
  • Will the feature overcomplicate things?
  • How much risk is there in this new feature?
  • How innovative is the new feature?
  • What do users say they want?

The Problem-Solution Canva

Should I Move to the Next Stage?

  • Are people using the product as expected? If not, (how) do they still extract value out of it?
  • What percentage of your users/customers are active (define active)?
  • Evaluate your feature roadmap against our seven questions to ask before building more features - does this change the priorities of feature development?
  • How do your user complaints impact feature development going forward?


Chapter 17 - Stage Three: Virality

"network-assisted word of mouth"

Focus on user acquisition and growth, but keep an eye on your stickiness too.

The Three Ways Things Spread

  • Inherent virality is built into the product, and happens as a function of use.
  • Artificial virality is forced, and often built into a reward system.
  • Word-of-mouth virality is the conversations generated by satisfied users, independent of your product or service.

Metrics for the Viral Phase - Viral Coefficient

Combined interest - not practical due to shared and limited circles of connections

Cycle time is paramount as well

How to push your viral coefficient to 1

  • Focus on increasing the acceptance rate.
  • Try to extend the lifetime of the customer so he has more time to invite people.
  • Try to shorten the cycle time for invitations to get growth faster.
  • Work on convincing customers to invite more people.

Beyond the Viral Coefficient - Net Promoter Score (NPS)

Instrumenting the Viral Pattern

Growth Hacking - Leading Indicator (something you know today that predicts tomorrow)

Attacking the Leading Indicator's

What Makes a Good Leading Indicator

  • social engagement or return frequency
  • tied to a part of the business model
  • coming early in the user's lifecycle or conversion funnel
  • early extrapolation

Causality Hacks the Future

Should I Move to the Next Stage?

  • Are you using any of the three types of virality (inherent, artificial, word of mouth) for your startup?
  • What's your viral coefficient? Even if it's below 1, do you feel like the virality that exists is good enough to help sustain growth and lower customer acquisition costs?
  • What's your viral cycle time? How could you speed it up?


Chapter 18 - Stage Four: Revenue

Metrics for the Revenue Stage


Chapter 19 - Stage Five: Scale

...


Chapter 20 - Model + Stage Drives the Metric You Track

...



PART III - Lines in the Sand


Chapter 21 - Am I Good Enough?

...


Chapter 22 - E-commerce: Lines in the Sand

...


Chapter 23 - SaaS: Lines in the Sand

...


Chapter 24 - Free Mobile App: Lines in the Sand

...


Chapter 25 - Media Site: Lines in the Sand

...


Chapter 26 - User-Generated Content: Lines in the Sand

...


Chapter 27 - Two-Sided Marketplaces: Lines in the Sand

...


Chapter 28 - What to Do When You Don't Have a Baseline

...



PART IV - Putting Lean Analytics to Work


Chapter 29 - Selling into Enterprise Markets

...


Chapter 30 - Learn fro Within - Intrapreneurs

...


Chapter 31 - Conclusion: Beyond Startups

...