editione1.0.8
Updated August 24, 2022You’re reading an excerpt of The Holloway Guide to Technical Recruiting and Hiring, a book by Osman (Ozzie) Osman and over 45 other contributors. It is the most authoritative resource on growing software engineering teams effectively, written by and for hiring managers, recruiters, interviewers, and candidates. Purchase the book to support the author and the ad-free Holloway reading experience. You get instant digital access, over 800 links and references, commentary and future updates, and a high-quality PDF download.
Good interview questions help sell the specific technical challenges of your team and ensure that you are looking at the right things in hiring. How do you get the signal you need and help to sell candidates on the priorities and competencies of the company?
Structured interviewing is the practice of applying the same assessment methods to review the competencies and traits of every candidate for a given role. This requires a calibrated set of interview questions that reviewers pose with consistency to candidates, as well as clear criteria for assessing candidates’ responses. In addition, interviewers must have familiarity with the question set and any associated expectations. Studies have shown that structured interviewing more effectively predicts job performance and is less prone to bias than letting interviewers casually decide what questions to ask.*
The purpose of structured interviewing is to improve the signal-to-noise ratio.
Noise is an incidental error that can distract from substantively useful information. Factors that can produce noise in an interview process include an interview starting off on the wrong foot, the candidate having seen a similar problem before, the interviewer’s mood, and so on. Any particular candidate’s performance assessment can vary from interview to interview, as demonstrated in a 2016 study by interviewing.io, and noise can account for many of those fluctuations.
Conducting multiple structured interviews can minimize noise. Google found that in their process, conducting up to four interviews increased their hiring accuracy—after four, accuracy increased, but only marginally.
Structured interviews often pull from question banks to ensure that all candidates are interviewed the same way.
A question bank is a collection of scripted interview topics and problems. An effective question bank also includes examples of expected answers and a rubric by which to assess candidates’ actual answers. Question banks will be tailored to reflect the nature of a company’s technical challenges and needs, and they take time to build.
caution Structured interviewing has a few downsides. If your questions are too rigid, you might not be able to effectively assess candidates with unique or uncommon skill sets. You can also acclimatize to your own questions, not realizing you have made them too difficult for most candidates. Lastly, every time a candidate performs better on an interview question than anyone you’ve ever seen, there’s a risk of letting that set a new standard that everyone else is judged against. Here, as always, rubrics help.
It is wise to be aware of the possibility that candidates or even teammates may leak the interview questions.
A leaked question is an interview topic or problem that is made available to a candidate before their interview. A candidate may have heard about the question online or from previous candidates. Leaked questions affect a company’s ability to fairly and effectively assess all candidates.
Repeatedly using the same questions increases the chance of leaks. There is a delicate balance between worrying about leaked questions too much and worrying about them too little. Most people won’t leak your questions, and if they do, it’ll likely be to a very narrow audience. You’re probably at higher risk of mis-signal in situations where a candidate has interviewed at a lot of companies at once, excels at whiteboard coding and storytelling, and maybe encountered similar questions elsewhere. That said, people who are good at spotting patterns and adapting like this are probably not the worst people to hire! Being good at ramping up on something is a useful skill for your employees. Nonetheless, with a little work, you can make your questions comparatively leak resistant.
First, rotate your questions. You can do this in a rolling fashion—that is, add a new question, test it for a while, remove the oldest question from the rotation, and repeat. This gives time to calibrate the rubric for the new question.
Second, leaks are less likely when you have questions with depth and nuance, rather than relying on questions that can be unlocked with a single trick; people who may be inclined to leak your questions likely can’t also leak every detail of the hardest parts. You can increase the depth your questions have by asking candidates to go beyond mechanical answers, with follow-up questions that require an understanding of underlying principles. Answers to the most subtle and nuanced questions can’t be faked—if the interviewee can navigate them, it means they really do have command of the material.
Bias is an observational error that tends to over-favor or under-favor certain types of candidates. As a systematic error, it is a common source of noise. For example, likability bias might cause an interviewer to view a friendly candidate as being more competent.
Bias constitutes a fundamental problem in hiring, and one key goal of interviewer training is to reduce the impact of interviewer biases on the final outcome. Bias detracts from interviewers’ ability to accurately assess candidates’ ability or fitness and thus increases the odds that you will hire the wrong people and create homogeneous teams.* Structured interviewing helps mitigate bias in interviews by using rubrics and requiring written justifications for decisions. Good training prepares interviewers to expect and detect the presence of bias in their own evaluations.