Picking good Product Managers - insightful interview questions
In early March I was in a Product Council for a Y-Combinator graduate company that really touched a nerve I’d like to write about: far too many product managers don’t really have a good foundation for what they do so they’re just not very good at it. So, how do you pick the good from the bad?
I’ve personally interviewed hundreds of product managers, and with the other members of the council we’ve collectively interviewed many many more. Towards the end of the session we discussed some of our interview questions for picking good product managers which I’ll summarize here, but first I need to cover some background: how product got here and why the discipline is still immature.
Y-Combinator Product Councils
The idea with the Product Councils is to give early stage companies access to advice from experienced product executives, specifically around their product related problems. The councils can be super impactful for companies as they navigate their growth / scale phase because they begin to experience the difficulties encountered by larger companies but they don’t yet have the resources or scale needed to recruit very experienced executives.
From my perspective though, they’re also an awesome opportunity because I get to:
See inside another company and gain better pattern recognition for common pains
Listen to how other product executives think about product related problems
In regards to the later I was joined in this session by truly top notch product people:
Chris Yeh - Former SVP of Product @ Box, currently CEO @ WhereTo
Charles Zedlewski - GM of Strategic Businesses @ Cloudera
Adrian McDermott - President of Products @ Zendesk
The company in this session had already been very successful with founders that were engineering trained but product oriented, along with a great head of design. They’d reached the point in scaling where they needed to add a Product Management function but weren’t sure exactly what to look for.
Building a product team - Picking good product people is really hard
Instead of a discussion of how the company could build a good product function, the council launched in to a warning around the state of the product management discipline and how to avoid bad product people. The group was really clear that hiring good product people is very hard and that the company would need to be careful.
It’s kind of weird that a bunch of product executives would begin a discussion around building a product team with warnings so stern that most people would turn back. Why is product in such a state?
Product is a deceptively complex discipline
Over the years I’ve been in product, I’ve been asked for advice by countless aspiring Product Managers. They’ve come from all sorts of backgrounds: technical, non-technical, sales, operations, engineering, you name it, I’ve seen it. Almost all of them have been a stakeholder working with a product manager, they’ve observed the mechanics of product: prioritization, requirements gathering, delivery management, progress communication, etc.; and felt like they’d be good at those things. Unfortunately that insight isn’t all that helpful.
The problem here is three-fold:
If your view of product is based purely on experience you may well have been observing bad product management, so there may be no transferrable lessons
Even if you’ve witnessed good product management, as a stakeholder you’ve only seen part of the playbook, and doing those things perfectly wouldn’t make you a good product manager
The mechanical parts of product management that are the most observable (sprint ceremonies, delivery management etc) are the least important parts since they focus on the what or the how, rather than the why and who
Good product management is defined by delivering truly game-changing experiences that meaningfully improve things for a group of real people. The mechanics of product appear to the outsider to be all about decisions but good decisions are driven by great discovery that is far less visible.
Bad product is worse than no product
Organizations look to product management teams to help them maximize the impact of R&D resources towards meaningful business outcomes. When there is no product management these decisions are normally made through the wisdom of the team members or through the insights of a founder / natural leader.
A product manager without a systematic approach to their discipline has less context than the team members so is less likely to make good decisions naturally and since they are specifically entrusted by the team to make informed decisions, usually has less healthy pushback to cause them to justify their decisions and examine their preconceptions. Thus a bad product manager is more likely to cause the team to make mistakes than no product manager at all.
The worst part is that many experienced product managers will be convincing since they’re used to influencing organizations. No matter how convincing the vision a product manager might paint, they need to be able to justify it within the competitive environment, the strengths of the organization, the data that proves the market exists etc.
Interview questions
As I listened to the potential interview questions that the council gave to the company it was obvious to me that they were all designed to uncover Product Managers who didn’t have a framework for their work and decisions. I really enjoyed them and have added them to my go-to questions.
Chris Yeh, CEO @ WhereTo, fmr SVP @ Box - “Can you please draw your product for me on the whiteboard?”
This is a super high level question. What does it mean to ‘draw a product’? Is it an architecture diagram, a market fit map, an OKR discussion or something else?
Even though it’s a very broad question, when Chris mentioned it I immediately thought back to the best candidates I’ve ever interviewed. Close to 100% of those involved the candidate (a) white-boarding, (b) diagramming the product and how it worked and (c) engaging me in a dialog of the market and how they’d win.
If you can’t explain your product in real-time and bring the interviewer on a journey you can’t really win in product. You must be able to:
Describe the market, the competitors, the macro and micro trends. Where is the puck going?
The product vision, and how this builds over time. How will you carve out space today and leverage that space in the future?
The qualitative and quantitative data that you’re using to orient yourself. How can you be sure you’re on the right track and how will you know you’re winning?
Influence people and bring them along with you. Why should I believe you could deliver this?
Charles Zedlewski, GM of Strategic Businesses @ Cloudera - “We’re debating investing in <X> or <Y>, which should we do?”
While Chris’ questions were focused in the context of the candidates’ previous product decisions, Charles’ question was designed to understand how a candidate would think through an important product decision that Cloudera was actually making.
It’s obviously unrealistic to imagine that a candidate would get to a better decision than Charles and his team could given the context differential, but the point of this question is to understand how they’d think through the question. Some signs someone knows what they’re talking about here:
They will explore the decision in terms of the customer base. Who does this decision impact? What data (qualitative and quantitative) exists that suggests the right answer for those users?
They’ll explore the decision in terms of the market. Where is the broader market headed? Does one path of this decision allow the company to take advantage of market tail winds? Are competitors headed in a specific direction that leads to a quantifiable opportunity in the other direction?
They’ll try to understand the business context. Is one path easier for the company? If so what’s driving that direction being the ‘path of least resistance’? Is there a weakness here that can begin to be remediated by taking a specific path? Does one path compound the strengths the company already has and thus have more leverage?
Adrian McDermott, President of Products @ Zendesk - “What is the ‘no’ answer you’re most proud of and why?”
With this question Adrian is driving at the key function of a Product Manager, focussing the companies limited development resources towards achieving maximum business benefit. To truly succeed at this you have to say no to a lot of good but less impactful ideas. In many cases the ideas you’re saying no to are suggested by powerful internal stakeholders or important customers, so you need to be able to justify those decisions with a clear framework for thinking about business value and excellent influencing / persuasion skills.
Terrible answers to this question will describe a politically charged decision where the PM asserted their ‘right to own the decision’ or just talk about how they said no to X feature and later turned out to be right.
As with Charles’ question, good answers to this question will describe the context for the decision and all of the nuance involved. After describing the context and how they decided to say no, good candidates will describe the stakeholder engagement tactics they used to say no while still strengthening their relationship with the requestor.
My favorite - “What are the common mistakes you observe less senior product people make?”
Chris, Charles and Adrian’s questions are a great way to dig in to how a candidate understands all of the different pieces and parts of product management. You want the candidate to demonstrate diversity of thought, i.e. be able to think though all of the pieces of the puzzle in a structured way.
This question that I always use is designed to draw out how Product Managers have developed over time and added more tools to their tool box. While the question refers to ‘other people’ the vast majority of candidates take it as a low risk opportunity to talk about their own learnings and developments as people. The framing makes it much less confronting than ‘tell me about your biggest weakness’ and other old chestnuts.
Good answers to this question tend to include the following:
Development from being sure of the answers because the candidate saw themselves as an expert to a system for structuring quantitative and qualitative data to find the answers
Increasing focus on the broader market opportunity and latent needs of potential customers versus focus on ‘our customers say X’
A trend towards viewing engineering and design as key partners to ‘getting to the right outcome’ rather than resources to be pushed to deliver
Increasing depth of thought on how to measure progress, specifically addressing metrics deeper than just revenue or total active users. This usually includes some approach to understanding success of features separate from success of the product as a whole
I love it when candidates mention books they’ve read, courses they’ve done, blogs they follow and other resources they’ve used as they talk through their development. Good product managers don’t learn everything from first principles, when faced with new problems they understand there’s a broader literature they can look to for answers.
Summing it up
Observing the major activities of product managers is not a great way to understand how good they are. Good product managers have great breadth: they understand the industry, the market, the trends and the customer base. They understand how to use a structured approach to collect quantitative and qualitative knowledge. They use a system to convert this knowledge in to good product decisions. They are avid learners, about the market, company and the teams they work with, but they don’t discover everything from first principles and they understand there’s a broad body of reference knowledge.
It’s critical that Product Managers can sell a story and sweep you away in a vision of a world you would like to believe in, but in product the goal is not to be right, it’s to have a system for reliably getting to right over and over again.
Thanks
My thanks to Chris Yeh (@cbyeh), Charles Zedlewski (@zedlewski) and Adrian McDermott (@amcdermo) for letting me share some of their wisdom in this essay