editione1.0.8
Updated August 24, 2022You’re reading an excerpt of The Holloway Guide to Technical Recruiting and Hiring, a book by Osman (Ozzie) Osman and over 45 other contributors. It is the most authoritative resource on growing software engineering teams effectively, written by and for hiring managers, recruiters, interviewers, and candidates. Purchase the book to support the author and the ad-free Holloway reading experience. You get instant digital access, over 800 links and references, commentary and future updates, and a high-quality PDF download.
Interview questions generally fit into three categories: coding questions, non-coding technical questions, and nontechnical questions.
Coding questions usually form the foundation of most technical interviews because all software engineers must be able to write code. Coding questions are also usually the easiest to administer and assess. But there are a variety of technical skills that cannot be evaluated with only a coding interview.
Coding questions are technical interview questions that require writing code, either on a whiteboard or in a hands-on format. Non-coding questions can be technical questions where coding is not involved, such as design or architectural questions, or generally nontechnical, like behavioral questions.
Many of the skills that you will need from senior engineering talent are most usefully assessed outside of a coding interview, including system design, technical strategy, relevant domain expertise, and nontechnical skills like leadership and mentorship.
Coding ability assessment is essential to the interview process. But there are many kinds of coding questions, and they have varying difficulty, style, and efficacy at measuring coding skills in ways that are representative of the challenges of the job.
It’s generally important that your question bank cover both basic and more advanced coding. If you have only questions that are sufficiently easy that you feel any strong engineer should be able to answer (FizzBuzz, for example), then you’re not going to get enough signal on the candidate’s problem-solving or ability to handle anything that is nontrivial. But if questions are mostly very hard, false negatives become common. Since coding questions can be noisy, you will typically want multiple interviewers and questions on your interview loop.
cautionFocusing exclusively on coding in the interview process is generally unwise. Doing so provides a poor candidate experience as well, because candidates want to show what they’ve done and what they know, not just their ability to solve toy problems.*
The spectrum of types of coding questions you can ask ranges from those that are heavily algorithm-focused (like a dynamic programming problem) to those that more closely resemble real-world scenarios. It’s best to match the kind of coding questions you ask to the kind of work you expect candidates to do; this practice gives you more useful signal and is more engaging for the candidate. It is not wise to include coding questions that are either math problems in disguise (unless mathematical skill is a job requirement) or that feel arbitrary. Even strong candidates may struggle with problems that feel completely artificial.
The one caveat is that if you are hiring recent graduates, you can reasonably expect that they have taken algorithm classes in college; testing them against those types of questions will give you some signal of how much they were able to retain. However, if you’re hiring for entry level roles and comparing people with CS degrees to bootcamp graduates or those who are self-taught, this approach will produce strongly biased results.
Almost all good coding questions will have a way to scale in difficulty so that every candidate can make at least some amount of progress. The wider the difficulty range of your question, the more you can distinguish great candidates from merely competent ones.
It’s possible that any candidate you test has seen leaked questions from your prior interviews. But what you think could be an indicator could also be completely benign—no problem is going to be new to everyone, and there’s almost always some chance that you will ask a question that someone has heard elsewhere. Making sure that the questions you ask reflect the skills required for the role will help you steer clear of generic, low-signal questions.
It’s generally best to let the candidate choose the programming language that they use for coding questions, unless language expertise is critical to the role. Strong engineers are usually able to pick up new languages quickly. But some roles do demand deep expertise in complex languages. In either case, it’s most effective to be clear with the candidate about their options and to encourage them to use the languages they are most comfortable with. If they choose a language that they think will impress you but that they do not use regularly, or if they choose one used by your company, they may underperform.
How demanding to be about language and library knowledge requires careful judgement. Unless it is a hands-on format, it’s important to remember that the candidate answering coding questions doesn’t have access to the resources they normally would. This means you can relax your expectations around language syntax and library functions. Whiteboard code will never compile, so don’t treat it like it has to.
On the other hand, it’s not OK if the candidate really doesn’t know the language itself. Someone using curly braces and semicolons in Python indicates that they have very little familiarity with the language. Someone forgetting a semicolon in C++ is not a big deal. However, the most frequent issue is for a candidate to get distracted by trying to remember something specific that has no bearing on the result of the interview. It’s usually fine to let a candidate make up library functions that are “close” to what the real library function is if their memory fails them—but equally, inventing “magical” library functions that solve too much of the problem is not a good indicator. In most cases, if someone would use a library in real life, it’s wise to let them use it in an interview.*
Some have questioned the necessity of intense coding interviews and suggest placing a greater focus on design questions, behavioral interviews, and other tests of knowledge.
On the one hand, coding questions in artificial environments under the pressure of an interviewer’s watchful eye can make people perform poorly, making for ineffective coding assessments. On the other hand, coding questions are meant to screen for necessary technical skills. There are people who simply cannot write code or who have only studied computer science and never really practiced engineering in work environments. Looking at experience alone isn’t always a good indicator of how the person will perform in the new role. A coding interview can help ensure that this person is able to do the work.
Another approach is to only ask relatively easy coding questions to verify that the candidate can code fluently, and then use other interview techniques to test other skills. Other resources like past work samples may be reasonable alternatives for measuring the person’s likelihood of success in the role.
Strictly no-code interviews may make sense for certain senior roles, where communication and high-level problem solving are more central to success, and actual coding is more limited. For some junior positions, it may be easier to teach code than it is to teach good communication, so if you’re planning on having the person learn a lot on the job, factors like problem-solving and collaboration skills might be more important. (There are training programs and coaches available to improve employees’ coding abilities in these situations, if there are not enough available resources to mentor them in-house).
story “The best no-code interviews I’ve seen involved diving into a candidate’s resume and talking about past projects—reaching the edge of their knowledge on the topics and probing one step further. It allows them to show off what they know and gives an accurate picture of what they don’t. It requires a very senior engineer to do it correctly, but it works beautifully.” —Laurie Barth, Staff Engineer, Gatsby
story “Dropbox has a pretty high bar for coding questions. It’s not entirely clear if, from first principles, you ought to design a hiring system this way, but it’s the system we understand and operate.” —Alex Allain, Engineering Director, Dropbox
Coding questions collect signal on a person’s ability as a programmer. The non-coding questions collect the majority of the other technical signals that help determine if the candidate has the experience relevant to the role and job level.
It’s best for more senior interviewers to ask non-coding questions and to spend time calibrating on how to evaluate the non-coding portion of interviews, because these interviews:
Require higher-level skills that cannot be evaluated by more junior engineers who lack them.
Require more judgment to assess a candidate’s choices in a context that the candidate is not fully familiar with.
Are simply more difficult to administer, as they are open-ended and require adapting follow-up questions to a wider range of circumstances than can come up in most coding interviews.
Non-coding interviews may also require more shadowing time to ramp up than coding questions, and they are often somewhat more company-specific, because companies assess and value non-coding technical skills differently.
These interviews break down into roughly three categories: those that assess what a candidate has done previously, those that assess specific non-coding technical and domain knowledge, and those that assess a candidate’s ability to think through new problems or situations.
story “If you use interviews for leveling, make sure that they give you enough information for that assessment. If a candidate whizzes through, but you assess them at a low level just because there wasn’t enough challenge to peg them higher, this is a problem.” —Laurie Barth, Staff Engineer, Gatsby
Interviews that assess previous technical work generally follow a rough progression:
The interviews start out with the big picture of a particular situation or project. This involves getting the facts and high-level context.
How well does the candidate understand and describe the business context, the problem that was addressed, and any critical constraints their team was dealing with?
Does the candidate understand the big-picture system architecture and design?
Can they identify what part of the system they worked on, what their own personal contribution was, and what their role was in relation to the rest of the team?
Next, the candidate can pick some specific elements they personally worked on to go deep into. Within those areas, the interviewer can ask the candidate to dig into the details of the system, focusing on decisions they made and why they made those decisions.
What would they do differently if they were working on this project now?
How would their decision have changed if some set of constraints had been different?
It’s common for candidates to talk about a system that they used or interacted with but did not personally build. It is the interviewer’s job to get the candidate to identify their own contribution to the system and then to go deep into the areas where they feel most comfortable assessing technical decisions. It is critical to understand the technical decisions that the candidate made, because the point of the interview is to assess their ability to design a system under real-world constraints.
As part of this, the interviewer can also seek the underlying reasons for decisions. The candidate will hopefully have a compelling reason for each choice, although that could be something pragmatic like “we had to move fast, and we had experience in this stack.” Not every decision needs major deliberation, but as an interviewer, you will be looking to see if the candidate understands every decision they made.
Additionally, because no system design is perfect—and the interviewer may in fact disagree with the design—the interviewer can then ask the candidate what they learned and would do differently. A strong candidate will be able to demonstrate their ability to identify problems, recognize when things didn’t go well, and iterate on a solution.
Finally, by asking the candidate how they would have changed their decision if some constraint had been different (for example, you have a month rather than a quarter to build the system), you get a chance to validate that they understand the decision well enough to adjust it in the face of new information.
caution All of these questions require that a trusted senior engineer exercise a tremendous amount of judgment and have the ability to pattern-match the challenges a candidate faced with their own experiences—which makes these probably the most difficult of all technical interviews to calibrate on.
Nontechnical interviews may also focus on assessing specific elements of a candidate’s technical knowledge. For example, you might ask them to explain a complex technical topic or ask questions about their knowledge within a particular domain. The former is similar to an assessment of past work, where you want to validate that the candidate understands the topic well by assessing underlying reasons for decisions. The latter is more about testing domain knowledge and can be used when filtering out candidates who lack fundamental knowledge in computer science or when specific domain expertise is relevant to a role.
In the third form of this interview, the interviewer asks the candidate to demonstrate a non-coding technical skill during the interview, so as to give insight into how they think through challenges. These interviews can include all sorts of relevant problems, like:
Designing a new system, architecture, or algorithm at a scope and domain appropriate to the role (sometimes called a System Design Interview).
Product thinking, such as handling ambiguous product specifications, prioritizing what to build, or evaluating trade-offs of implementing different features.
These interviews may be easier to calibrate than ones that evaluate a candidate’s prior work because the problem-solving follows a more consistent flow, and you can write very clear rubrics.
caution At the same time, beware of groupthink and rigidity in assessments. Senior engineers may be wedded to specific architectural patterns, for example, and they may dock those with a different perspective. But there is immense value in having someone come into your organization with fresh perspectives, especially at the senior level.
One skill that you will expect candidates to demonstrate is the ability to handle more open-ended problems, so it’s wise to frame them accordingly, especially for more senior levels.
| Prompt | Evaluation |
|---|---|
| Imagine we’ve gone back in time and we’re about to build Acme product again from scratch. We want to get to a prototype within four weeks. It doesn’t have to be perfect, but it needs to have the key parts in place. Can you take five to ten minutes to flesh out for me how you’d approach this? | Has the candidate thought through the product and the key features ahead of time? Are they able to draw a sensible diagram of the major services they would need to have in place and the interfaces by which they might communicate? Are they able to arrive at a decent schema for the service? Extra points if they realize any of the complexities (for example, comments might require editing history to be stored). |
| One of the things about our service is that it pulls from APIs like Twitter and AngelList. They have API quotas that prevent us from pulling in all the data we want. Can you walk me through how we might design a system for prioritizing the most important calls? | Is the solution adaptable to future third-party services? Is the solution adaptable to ongoing changes in API rate limits/quotas? Do they have examples where they’ve worked backward from what the customer is going to care about into their solution? |
danger There are some interview questions that are designed to see how the candidate reacts to the parameters of the question, like asking an impossible question, never giving enough guidance on what the question is really about for the candidate to succeed, or being incredibly silent or non-interactive during the interview. Anything that feels like it could even possibly be considered a mind game is bad territory to be in.
Not only is this a poor way to treat a candidate (and strong candidates with other options will rightly drop out if treated like this), but an interview must assess candidate-company fit for both sides. Candidates want and need to know how you work and communicate, and it’s your job to demonstrate that to them.
If you go into an interview with the intention of lording your knowledge over a candidate, showing them how smart you are, they can tell. And if you ask questions but don’t really listen to the answers, it’s all too obvious.Eric Ries, author of The Lean Startup*
danger It’s critical not to fall into the trap of trying to compete with the interviewee or to impress them for the sake of the interviewer’s ego.
A shibboleth question is an interview question that seeks to quickly assess a candidate’s general capabilities in an area by asking a question that the interviewer believes anyone in that area ought to know the answer to.
dangerThese questions are dangerous because not everyone will agree on what the common knowledge base in an area is, and asking for very specific knowledge will filter out qualified candidates. For example, not all software engineers will necessarily know dynamic programming.
These kinds of questions can be used by technical screeners and even recruiters if you have a high enough volume of candidates that you need some way to filter and also can’t find another way to distinguish otherwise equally qualified candidates. But it’s risky and does not create a good candidate experience, because it suggests you care more about factual knowledge than general ability, which rubs people the wrong way. It’s also problematic for candidates from nontraditional backgrounds. Anything keyword-based, especially when assessed by a non-coder, is going to be biased against them.
cautionTwo other specific kinds of questions to watch out for are brainteasers and Fermi problems. These kinds of questions used to be popular, but neither have been shown to correlate with job performance and may annoy many candidates or distract from more practical and relevant signals.
Because engineering does not take place in isolation, everyone you hire must be able to work well with other people. As your organization grows, and as individuals become more senior, the nontechnical aspects of the job become increasingly important.
There are several ways to evaluate nontechnical skills, including behavioral questions, situational questions, and role-playing. It is critical to anchor your assessment of nontechnical skills in your defined company values, so as to ensure you are hiring people with the skills that map to the expectations you will have for them once they join. Just Googling “icebreaker questions” or asking candidates to “tell me about your hobbies” will not produce results that will serve you well. It’s important that nontechnical interviews be just as serious and structured as every other aspect of the interview process.
As discussed in Preparing Interviewers, it may make the most sense to have senior people conduct nontechnical interviews, which are typically harder to assess than straight coding interviews. Some managers recommend conducting all nontechnical interviews as pair interviews, including one senior person and one less senior person. This can help to reduce bias when assessing subjective answers and helps to train junior interviewers in a kind of active shadowing. (Recall that you may have to adjust for the possible pitfalls of having different-level pairs.)