Here’s a repeatable process for turning an interview, or a whole interview loop, into a hiring decision. This is a good process to start with when the whole hiring team can no longer fit in a single large conference room.
TL;DR
Interviewers refine raw notes into smaller sets of stronger signals, until the hiring decision feels obvious. A bar-raiser does the same thing with the feedback from each interview. And the bar-raiser and hiring manager make a final decision.
Post-interview stages
This process starts at the point where all interviews (aka interview loop) have been conducted. It ends at the point that the bar-raiser and hiring manager have decided to continue with hiring the candidate - usually moving to reference checks - or reject them.
- Each interviewer writes standard-structured feedback.
- A bar-raiser (or hiring manager if you don’t have bar-raisers) summarizes the feedback across interviews.
- Everybody gets together in a debrief; the hiring manager and bar-raiser make the decision with assistance from interviewers.
Interview feedback
After an interview, the interviewer writes feedback in this format.
I’m {inclined/not inclined} to hire {Candidate} for {Role}
Top reason to hire: One sentence about skill or attribute and how Candidate demonstrated it
Top reason not to hire: One sentence about skill or attribute gap, and how Candidate demonstrated it
Optional prose explaining the interview’s highlights/lowlights, with 1-2 sentences explaining key positive and negative points
Bullet points beginning with
+or-, in this form+ {Company value or job skill}: {a few words about how candidate demonstrated this} + {Company value or job skill}: {a few words about how candidate demonstrated this} - {Company value or job skill}: {a few words about how candidate demonstrated this}
Each section is a conclusion. And each conclusion comes from the following section’s data. This makes it easy to spot conclusions that don’t follow the data, and which stage of the pipeline has an error.
Feedback is written from most to least granular (so, bottom of feedback to top), because it’s easiest to follow and debug your own analysis. It’s also easier to show work to others.
- raw interview notes
- signals
- key signals
- decision
What’s a signal?
Signals are company values, technical skills, or coworker qualities that the team/company have agreed are important, and the candidate demonstrated having or missing.
Some signals from technical interviews I’ve done in the past are:
- [General competence] commmunication, love problems not solutions (a company value), resolving ambiguity, project planning
- [Technical skills] coding, architecture, debugging, use of metrics
- [Teamwork skills] humility, mentorship, resolving conflict
Examples from real interviews
+ humble: one specific career failure, which was a toxic hire. "at amazon I argued for [good hiring]... [and then] I did something really stupid"
- effectively setting context: I had to probe to understand some of the product requirements (subscriber vs free model)
+ ability to collaborate: Very focused on solving customer problems; worked closely with them to define project
A quick note on bias: Bias in interviews is a whole other blog post, or a book. A quick way to remove bias based on the candidate’s name or gender is to use the words Candidate (C for short), or the pronouns they/them.
Which signals are important?
This is a hard question to answer; my instinct is that that’s a function of lots of interview experience, and experience working with lots of different types of people.
It’s also specific to the role, level, team, and company. But broadly speaking, a key signal should be something that’s (a) high impact on the candidate’s performance and (b) you are confident you heard or saw in the interview.
Examples from real interviews
key reason to hire: C raises the bar on being self-aware and intentional on how they interact with + grow their teammates
key reason to hire: Hunger/Urgency. C was very focused on customer impact, delivering value quickly
key reason not to hire: C was entirely focused on technology and implementation, and pretty unaware of people skills or business prioritization. This isn't uncommon for junior candidates, but would impose a hard limit on scope of impact
The debrief doc
The debrief doc format is very similar to the single interview format, but begins with details and works down to analysis.
+ {Company value or job skill}: {a few words about how candidate demonstrated this} + {Company value or job skill}: {a few words about how candidate demonstrated this} - {Company value or job skill}: {a few words about how candidate demonstrated this}Top reason(s) to hire: One to two sentences about key positive skills or attributes demonstrated across interviews
Top reason(s) not to hire: One to two sentences about key missing skills or attributes, and how the gaps were demonstrated
Recommendation: {hire/no hire}
Writing a debrief doc
Reading multiple interviews’ feedback in the standard structure makes it straightforward to spot common themes across interviews. This can be done rapidly with practice. 5 minutes is about the fastest I could ever manage, but 15-20 minutes was probably the norm.
- I start by breaking pros and cons into technical and team player categories, and writing down any repeated signals, or particularly strong signals from individual interviews.
- If doing this in a rich-text format, I bold the strongest signals. If not, I prefix the positives with lots of
+s, and negatives with-s. - I write a preliminary hire recommendation for review later during the debrief.
Making a hiring decision
The decision, made by a bar-raiser and hiring manager in concert, should not be averaging everybody’s individual decisions. It’s okay to hire somebody who had an individual interview go poorly, or who didn’t have universal hire recommendations. And it’s okay to not hire someone who had a strong interview or two.
But using these common feedback structures, it’s much easier to spot significant strengths or risks that no single interviewer could spot as a deciding factor.
Why this process works
This process is designed to force interviewers to spend a little time reviewing and thinking about the interview, in order to write down the signals they got and the second-stage analysis of those signals.
It’s a great channel for making bias obvious and removing it from the process.
It prevents the loudest interviewers from dominating the debrief (and thus, hiring decision).
And it forces bar-raisers and hiring managers to explicitly and consciously make tradeoffs.
Learning from others
A nice intended side effect of this process is that this process makes it easier for interviewers to learn from others. Interviewers reading individual feedback summaries and the debrief doc learn
- What skills or values the team cares about the most, and which are considered coachable or nice-to-have
- How other interviewers interpret candidate performance
When is this not a good process?
I don’t think I’d adopt a process this rigorous, early on in a team’s hiring journey. If the whole company is under 50 people, you might still be figuring out your roles and responsibilities, what levels look like, etc.
I also don’t think it’s required at very low hiring scale. If individual interviewers only do a few interviews a year, and you’re hiring 10 people a year or less, it might not be worth the training cost of adopting this.
Appendix A: FAQ
Why not just share raw notes?
Raw transcript notes (which I write, myself) are difficult for the next person to read. They also don’t show which moments in the interview were critical. Without summarizing the interview, the interviewer may still try to share this, but it’ll be slow, wasting the time of the rest of the interviewers.
Why re-write feedback as a debrief doc?
Same as above. By spending a few minutes reading and summarizing, the bar-raiser can save everybody reading and analysis time, meaning the debrief itself if more focused on if the right tradeoff was captured, and whether it’s a good one.
Why use a bar-raiser?
I think this process can work without one, where the hiring manager does the summarization across the whole loop. I think a bar-raiser is more effective though, because without having done their own interview of the candidate, they don’t have a pony in the race and find it easier to be unbiased.
It’s also a great way to deploy your most experienced interviewers; they end up teaching the whole interview loop at a time instead of just having one person shadow them at a time in an individual interview.
Why make a hiring decision as an interviewer if someone else is going to redo that step?
Interviewers could probably skip this without too much of a problem. I think it’s worth doing though, because (a) it helps teach interviewers how to make intentional tradeoffs about people, and (b) it’s actually very little work once you’ve done all the other analysis!
Appendix B: Origin
The process above is almost exactly what the engineering team at Convoy used from 2018 onward.
Our first debrief process was simpler for the first 3 or 4 years.
- Written feedback was encouraged but not enforced.
- We’d spend the first 20 minutes going around the room, asking individual interviewers to tell us what went well/poorly from their interview, and writing lists on the whiteboard.
- We’d all try to look at these lists and the bar-raiser and hiring manager would come to some consensus.
There were obvious problems with that…
- Without requiring written feedback ahead of time, some interviewers would walk into the debrief without any notes
- Interviewers sometimes hadn’t spent any time between leaving the interview and the debrief thinking about pros/cons, so they’d spend 10-15 minutes of the 30 minute debrief trying to analyze and share with the group.
- With giant lists of
+s and-s, it wasn’t clear what the key factors for the hire decision were, even after it had been made. - The only way for interviewers to get feedback from more experienced interviewers or the bar-raiser was immediately after the debrief, assuming they both were available and the BR had been able to scribble down what they wanted to share.
- It was inconsistent what problems were acceptable or deal-breakers
- It wasn’t clear which unusual strengths were
This new process added a debrief doc, requiring written feedback ahead of time, and reviewing the doc during the debrief. It was trialed side-by-side with the old process by several of our strongest interviewers. And I remember it being immediately well-received by the bar-raiser group and adopted pretty painlessly.
The only thing we never perfected was standardizing all interviewers on the interview format. That made writing debrief docs occasionally difficult. Sometimes interviewers would paste a giant transcript with no summary and the BR would have to read and summarize the interview. A less painful but more common problem was a 15-point list of signals without any indication which signal was strongest.
Appendix C: Bar-raisers
This is a concept originally created at Amazon and used somewhat widely across the tech industry - especially by people who spent their formative years there :)
The idea is that you get somebody who’s outside of the hiring team, and thus less biased, to make the call on whether this is a hire to fill a seat, or actually meets the org-wide or company-wide bar.
As far as I know, Google does “hiring committees,” a group of people who don’t meet the candidate and review written feedback from the actual interviewers and make a hiring decision based on written feedback alone.