The Ultimate Guide to Cracking Product Case Interviews for Data Scientists (Part 1)

Many people find product case (also known as business case) interviews difficult. However, for data scientists having a strong product sense and being able to help stakeholders make data-driven decisions is crucial.

That’s why in this post were going to summarize what you need to know to crack product case interviews with confidence. This is a two-part post, and in this first part, we will be going over the basics of product case interviews and metric frameworks.

Product Case Interview Basics

You can expect to encounter product case questions during both the technical phone screen and on-site interview. Product Analytics Data Scientists will likely encounter more of these questions than other roles.

These questions are focused on a particular business situation or case (hence the name). You will be asked about things like how to improve a product or how to measure the success of an idea. There usually is not one correct answer because the questions are open-ended.

Even though they might not be looking for a particular answer, interviewers are still checking for certain qualities that make a good answer. Your answers need to be…

• Structured showing a systematic approach to the problem.
• Comprehensive covering the important but not necessarily all aspects of the problem.
• Feasible representing a practical solution.

Larger companies may expect candidates to demonstrate familiarity with their products as well.

Who your interviewer is can also impact what the interviewer wants to see. For example…

• Individual contributors tend to put more emphasis on technical expertise.
• Data science managers are interested in your ability to communicate and drive decision-making.
• Product managers value candidates who can think practically and strategically about the bigger picture.

The Product Development Process

Before we get into more specifics about product case interviews, it pays to understand the product development process. That’s because data scientists play a critical role in the decision-making in this process, and the questions you get in a product case interview will be centered around the stages of the process.

Product development can be broken down into 4 stages (the bullet points show what data scientists typically do in each stage):

1. Coming up with initial product ideas.
2. Selecting ideas:
   • Quantitative analysis to select a subset of ideas to which to devote resources, often referred to as opportunity sizing.
3. Experiment design:
   • Involved with selecting success and guardrail metrics, running sanity checks, choosing randomization units, etc.
   • Candidates will need to consider alternatives when it is not possible to run A/B tests.
4. Making a launch decision:
   • Making scientific decisions based on experimentation results.
   • Diagnosing problems and evaluating tradeoffs.

At each stage in the process using the appropriate metrics is vital. In fact, metrics are the most important component of product case interviews for data scientists, so now let’s turn our attention to them.

Metrics

The first thing you need to know about metrics is how to classify them. The taxonomy of metrics allows us to know what metrics are used for what.

• Success metrics are used to measure the success or health of a product.

  Examples:

  • Daily/monthly active users
  • Bookings/Purchases
  • Revenue

• Guardrail metrics measure core quantities that should not degrade in pursuit of a new product or feature.

  Examples:

  • Bounce Rate: cancellation/unsubscription rate, latency, etc.
  • Success metrics of other products

Besides knowing the types of metrics, you also need to know what makes a metric good. Metrics should have three qualities:

• Simple
• Clear
• Actionable

There are several “metrics frameworks” that cover the vast majority of business cases. Let’s start by looking at two general frameworks.

General Funnel Metric Frameworks

A funnel metric framework is a family of metrics that tracks the “user journey” through various parts of a product.

Examples of funnel metric frameworks and the metrics included are:

• AARRR growth metrics framework (see details in Growth Metrics section below)
• Conversion funnel
  • Number of visitors to webpage;
  • Number of logged in users;
  • Number of users who click particular parts of the logged in pages;
  • Number of users who visit the checkout page;
  • Number of users who purchase.
• A B2B funnel
  • Number of visitors to webpage (leads);
  • Number of leads who request free trials;
  • Number of leads to which the Sales team proactively reaches out;
  • Number of paid customers (each of which is a company / business) who made recurrent purchases.

Funnel metric frameworks are particularly favored by business case interviewers. They make it easy for the interviewer to ask questions and probe deeper.

Input-Output Metrics Frameworks

These frameworks revolve around two key concepts:

• Input/driver metrics: Metrics that track the activities/resources used to work towards an outcome;
• Output metrics: Metrics that demonstrate the outcome of an initiative.

This framework is not often asked about directly, but it can give helpful structure to otherwise unorganized thoughts.

Besides general metric frameworks, there are also domain specific frameworks. Knowing these will give you a better understanding of how these metrics work in the real world.

Growth Metrics (AARRR)

This framework measures a product’s user base and engagement.

• Acquisition: Getting customers to sign up for a website or product.
• Activation: Getting customers to gain basic familiarity with the product.
• Retention: Getting customers to come back to the product on a regular basis; a customer that exhibits no (or minimal) activity over some predetermined time period is known as churned.
• Referral: Getting customers to share the product with others.
• Revenue: Getting customers to adopt one or more paid features that generate revenue for the company.

There are several things to think about within this framework:

• Tradeoffs between acquisition and revenue
• Different levels of engagement
• Retention and churn cannot be easily estimated over the short term
• It is possible for a churned customer to recover (often known as resurrection)
• Difficulty of measuring incrementality for referral initiatives
• Differences between B2B and B2C settings
• Challenges of international expansion

Platform Metrics (Customer Support, Trust and Safety, Payments, Infrastructure, Operations, etc.)

“Platform teams” carry out operational support (related to customer help centers, payment flows, fraud detection, infrastructure, etc.). The metrics of platform teams include:

• Costs of operations/infrastructure
• Success/failure rates
• False positive/false negative/true positive rates
• Fraud loss metrics
• Vendor costs

Conclusion

Remember that there are other business domains, like Finance, Pricing, Logistics and Supply Chains, Marketing and Sales, and Search Ranking, each with its own metric frameworks. We recommend exploring further on your own!

That’s all for Part 1! Be sure to check out Part 2 for more on what you need to know for product case interviews.

Also, check out the longer version of this post for even more detail.