Training Interviewers

You’re reading an excerpt of The Holloway Guide to Technical Recruiting and Hiring, a book by Osman (Ozzie) Osman and over 45 other contributors. It is the most authoritative resource on growing software engineering teams effectively, written by and for hiring managers, recruiters, interviewers, and candidates. Purchase the book to support the author and the ad-free Holloway reading experience. You get instant digital access, over 800 links and references, commentary and future updates, and a high-quality PDF download.

​story​ “It’s ridiculous to expect engineers to be competent interviewers with little to no training. To the extent you can be good at interviewing, it comes from repetition, learning the failure modes and bolstering against those. Experience just makes you much better at navigating the myriad situations. If this is your first remote interview, you’re not going to be able to tell if it’s bad because of the candidate or because it’s remote. Training often doesn’t exist, or it’s done really poorly. If you can’t invest in training engineers to interview, you can’t expect to get any real signal on candidates.” —Scott Woody, former Director of Engineering, Dropbox

Effective interviewing is a learned skill. It requires a mix of technical knowledge, emotional intelligence, and thinking on your feet, while being fair and rational. None of this comes easily, and doing it all at once is really difficult. Having experienced employees who have done interviews elsewhere go through your organization’s interviewer training will help ensure alignment.

Calibration is the process of developing the ability to accurately assess whether a candidate will succeed in a role. A calibrated interviewer will not only be able to assess a candidate’s interview performance but draw an informed conclusion about whether the candidate should be hired for a given role.

​danger​Candidates may have poor experiences if they are exposed to anyone involved in the interview process who is interviewing for the first time or who is not prepared to assess with respect to the specific role. Whether it’s an interviewer, hiring manager, or hiring committee member, without calibration, they will be unable to make a data-based recommendation.

​story​ “Uncalibrated people are not aware of common biases. It’s as much about self-awareness as it is about training. Broadly, engineers are pretty good at sussing out whether a person is like them, even if they’re not particularly trained. Engineers can be trained to be more flexible in who they can evaluate. Backend engineers self-identify as being unable to evaluate frontend. That’s common. So how are they going to bridge that gap? If they’re trained, they can.” —Scott Woody, former Director of Engineering, Dropbox

Calibration occurs through a mix of:

• Interview training and shadow interviewing.

• Question banks and structured rubrics that specify what level of performance is expected on different questions. Good rubrics draw on the collective experience of seasoned interviewers and recruiters.

• Mock internal interviews. You can train new interviewers and test new questions by mock interviewing candidates who are already on your team.

• Giving data and feedback to interviewers, like how their ratings for candidates compare to other interviewers’ ratings and how predictive their ratings are. This can help interviewers detect things like whether they are too lenient or too strict or are otherwise highly inconsistent with other interviewers.

We don’t just look at the candidate side of hiring. Interviewers also receive feedback on their own personal ability to predict whether someone should be hired. Every interviewer sees a record of the interview scores they have given in the past and whether those people were hired or not.Laszlo Bock, former SVP of People Operations, Google*

Interviewers and decision-makers perpetually improve their calibration as they gain more experience and interview more candidates, so no one is ever “fully calibrated.” But setting some minimum level of calibration helps any person be equipped to draw effective conclusions as they assess a candidate. For example, it helps to know whether the hiring team considers a particular question easy or hard, or whether a particular type of mistake should disqualify a candidate or not.

​danger​If your only source of calibration is the candidates you’ve seen, you could be at risk of incorrect calibration. This is especially likely to happen with new roles or new interview questions, where your company lacks a baseline. It is also likely if you are letting weak candidates into your funnel, because interviewers might find their perceptions shifting over time. They might then be overly impressed by a candidate who is “above average” (without realizing that the “average” is really poor for this particular search). This is sometimes referred to as “hiring the tallest of the bunch.”

One step to help counter this is to incorporate recordings of real-life interviews into your calibration process; interviewing.io published a few mock interviews on their platform.

Align on the Process and Role

It’s helpful to provide a general presentation or discussion of interviewing to everyone who will be involved. This briefing can be delivered by a hiring manager, an experienced interviewer, or a recruiter and can include:

• Why hiring (and interviewing in particular) is so crucial. Your team may be spending valuable time on interviewing candidates, and it’s important that they view it as an important responsibility (and not a chore or a waste of time).

• The existence and risks of unconscious bias and how it can affect judgments, as well as strategies for mitigating bias.

• The benefits of structured interviewing.

• What their role is as interviewers, and how that fits into the hiring process at large.

• What the interview process will look like.

• The fact that interviewers function as team ambassadors, and that in addition to assessing candidates, their actions may encourage or discourage candidates from joining the company.

• How to write effective interview feedback.

• What the interviewers’ role will be in the hiring decision. For instance, will they need to participate in debrief or huddle sessions after an interview? This information sets the right expectations about how hiring decisions will be made.

• The legal considerations for interviewers, including what questions the law restricts interviewers from asking.

Chuck Groom, Director of Engineering at Truva, provides a useful overview of the interview process that provides additional perspective.

An important part of interview training is ensuring interviewers understand the importance of their role in building a team. Keith Adams, Chief Architect at Slack, once interviewed a candidate who greatly impressed him: “I rated him a strong hire.” But during the interview debrief, it turned out that the candidate hadn’t impressed the rest of the interviewers. Keith made a passionate case anyway, and Facebook hired the candidate, who went on to have massive impact at the company. “To any engineer, interviewing can feel like a 45-minute interruption to your immediate work. But the people you help hire are a huge part of your legacy. Hiring that candidate was the most important thing I did that year,” Keith said. (Keith’s bar for impactful work is pretty high.)

Next, interviewers benefit from having information specific to each role or type of interview. What are the roles they might help interview for? What types of interviews will they conduct, and what primary competencies will they be assessing? Are there secondary things they may be looking for as well (maybe they are assessing technical skills, but should still note arrogance in a candidate)? Are there predetermined, structural interview questions to follow?

Occasional uncomfortable interactions may help make an interview more functional. We are taught to never interrupt, yet it’s not productive to let a candidate go down the wrong conversational rabbit hole; it’s appropriate instead to intervene and steer the discussion in a more helpful direction. This can be even harder to do with behavioral interviews. Most people are very good at talking at a high level about their experience but don’t go into any significant depth. It’s the interviewer’s job to help the candidate tell their story, which might mean cutting them off when they go off course and knowing when to steer more and when to steer less, which comes with time and experience.

​story​ “When I start an interview, I often tell the candidate I want to get certain kinds of signals. This means I might cut off certain directions of discussions or lines of thought to help with that. It’s not me being rude; it’s just me trying to make the best use of our time.” —Scott Woody, former Director of Engineering, Dropbox

A typical interview process may include four to eight interviews, where each interview is between 45 and 60 minutes. That means there’s never enough time to get every data point you’d like, and because interviews provide noisy signal, you can only have so much confidence in the information you collect. Therefore, it’s important to prioritize both for what really matters to you and what you can reasonably assess.

​caution​An interviewer may not like the style or “attitude” of a candidate’s response to a question. The candidate may deflect, go on a tangent, or even refuse to answer. This definitely means something is awry, but the cause may not be what you expect. On the interviewer or company side, an excessively junior interviewer may ask a question in a way that doesn’t suit a senior candidate; it could be a poorly designed question; or possibly the hiring team failed to convey to the candidate accurate expectations for the interview. On the candidate side, personality style or inadequate technical ability may result in an unwillingness to dive into technical material, even in response to a well-phrased question. The best solution is for interviewers to ask themselves, “What is the signal we want to get here?” and to focus on that. As an interviewer, you help avoid generating such candidate responses by knowing your role in the interviewing process, what signal you want to get, and why you’re asking these specific questions.

A good alignment process will usefully communicate to interviewers the scope and nature of the role:

The scope of the role is a reflection of how a role fits into an organization. Scope often aligns to a particular rung of a career ladder and conveys how much responsibility the candidate will be trusted to handle. Senior members of an organization can best discern whether a candidate is well suited to the scope of the role, especially in interviews designed to assess technical skills beyond coding and nontechnical skills.

The nature of the role is a reflection of the specific technical skills that the role requires. Individuals directly familiar with the work and the challenges facing the team can best discern whether a candidate is well suited to the nature of the role.

By way of reducing noise, Ammon Bartram at TripleByte suggests looking for signal on max skill, not average or minimal skill (or on where a candidate may make mistakes). This approach focuses on “looking for strong reasons to say yes and not worrying so much about technical areas where the candidate was weak.”

For an in-depth look at collecting signal, David Anderson’s thoughts on hiring for leadership roles at Amazon include a host of specific questions that build on each other, a practice that TripleByte also encourages.

Mock Interviews

Mock interviews are educational simulations in which individuals act out the roles of interviewer and candidate. Successful mock interviews resemble real interviews as closely as possible. Either potential candidates or potential interviewers can use them to prepare for the real thing. Companies may film mock interviews for the benefit of other trainees when preparing potential interviewers. Best practice includes following an observed mock interview session—whether live or filmed—with a debrief with senior interviewers who discuss what was successful and unsuccessful in the exercise.

Within a technical organization, mock interviews help junior engineers practice their interviewing skills and start to recognize what makes an interview successful or unsuccessful, focusing both on interviewer behavior and on how candidates navigate questions. This means that the person playing the candidate ideally will embody typical responses to the question, both successful and unsuccessful. Debriefs then focus on any pitfalls, patterns, or typical responses, and highlight what success looks like.

Behavioral interviewer training can be more difficult, and practicing with co-workers can be extremely helpful. Trainees practice how to probe into answers deeply, helping to build confidence around steering the candidate. The art is in knowing when to steer, and when you’re steering so much that you’re making the interview too easy—this comes with time and experience, and mock interviews simulate that.

Shadow Interviews

Shadow interviews are the next step. They provide crucial preparation for less experienced interviewers.

A shadow interview is an opportunity for a new interviewer (the shadow) to observe a more experienced interviewer (the lead) working with a candidate. Later, a trainee may lead an interview with a more experienced interviewer shadowing them in a reverse shadow interview.

In effective training, the shadow takes full part in the non-evaluative parts of the interview, including introductions, selling the candidate on the company, and answering the candidate’s questions, and remains attentive during the remainder.

​important​ Having two interviewers present might seem awkward or intimidating to candidates. Just as with pair interviewing, it’s important to prepare the candidate by letting them know there will be a shadow present, and that the purpose is to help train interviewers and provide a consistent experience for interviewees. Both interviewers can introduce themselves, and then the lead can clarify that one interviewer will conduct the interview and the other will mostly observe. Explaining that the shadow interviewer isn’t expected to say much (if anything) will help make sure the candidate doesn’t misinterpret their behavior as silent judgment.

After the shadow, the lead and shadow debrief by talking through what they saw, having the trainee share their thoughts about the candidate, and discussing the nuances of the question flow. A trainee might do this five or ten times and then be ready for the reverse shadow interview, with the more experienced interviewer providing one-on-one feedback afterward. Successfully completing the shadow interviewing process will prepare a new interviewer to go solo.

Many companies use similar processes, but Google has published their own shadowing process as part of an extensive online resource for training interviewers, with the caveat that it is designed in line with their own, atypical interviewing philosophy and infrastructure (for example, they use independent hiring committees, which may not be practical for other organizations).

​contribute​ We’re looking for more great interview training resources. If you know of any—or if there’s more you’d like us to write on the topic—please let us know!

If you’re interested in improving as an interviewer, read Alex Allain’s work on the Holloway blog.