Coding Questions

You’re reading an excerpt of The Holloway Guide to Technical Recruiting and Hiring, a book by Osman (Ozzie) Osman and over 45 other contributors. It is the most authoritative resource on growing software engineering teams effectively, written by and for hiring managers, recruiters, interviewers, and candidates. Purchase the book to support the author and the ad-free Holloway reading experience. You get instant digital access, over 800 links and references, commentary and future updates, and a high-quality PDF download.

Coding ability assessment is essential to the interview process. But there are many kinds of coding questions, and they have varying difficulty, style, and efficacy at measuring coding skills in ways that are representative of the challenges of the job.

It’s generally important that your question bank cover both basic and more advanced coding. If you have only questions that are sufficiently easy that you feel any strong engineer should be able to answer (FizzBuzz, for example), then you’re not going to get enough signal on the candidate’s problem-solving or ability to handle anything that is nontrivial. But if questions are mostly very hard, false negatives become common. Since coding questions can be noisy, you will typically want multiple interviewers and questions on your interview loop.

​caution​Focusing exclusively on coding in the interview process is generally unwise. Doing so provides a poor candidate experience as well, because candidates want to show what they’ve done and what they know, not just their ability to solve toy problems.*

Kinds of Coding Questions

The spectrum of types of coding questions you can ask ranges from those that are heavily algorithm-focused (like a dynamic programming problem) to those that more closely resemble real-world scenarios. It’s best to match the kind of coding questions you ask to the kind of work you expect candidates to do; this practice gives you more useful signal and is more engaging for the candidate. It is not wise to include coding questions that are either math problems in disguise (unless mathematical skill is a job requirement) or that feel arbitrary. Even strong candidates may struggle with problems that feel completely artificial.

The one caveat is that if you are hiring recent graduates, you can reasonably expect that they have taken algorithm classes in college; testing them against those types of questions will give you some signal of how much they were able to retain. However, if you’re hiring for entry level roles and comparing people with CS degrees to bootcamp graduates or those who are self-taught, this approach will produce strongly biased results.

Almost all good coding questions will have a way to scale in difficulty so that every candidate can make at least some amount of progress. The wider the difficulty range of your question, the more you can distinguish great candidates from merely competent ones.

It’s possible that any candidate you test has seen leaked questions from your prior interviews. But what you think could be an indicator could also be completely benign—no problem is going to be new to everyone, and there’s almost always some chance that you will ask a question that someone has heard elsewhere. Making sure that the questions you ask reflect the skills required for the role will help you steer clear of generic, low-signal questions.

Choosing Languages

It’s generally best to let the candidate choose the programming language that they use for coding questions, unless language expertise is critical to the role. Strong engineers are usually able to pick up new languages quickly. But some roles do demand deep expertise in complex languages. In either case, it’s most effective to be clear with the candidate about their options and to encourage them to use the languages they are most comfortable with. If they choose a language that they think will impress you but that they do not use regularly, or if they choose one used by your company, they may underperform.

How demanding to be about language and library knowledge requires careful judgement. Unless it is a hands-on format, it’s important to remember that the candidate answering coding questions doesn’t have access to the resources they normally would. This means you can relax your expectations around language syntax and library functions. Whiteboard code will never compile, so don’t treat it like it has to.

On the other hand, it’s not OK if the candidate really doesn’t know the language itself. Someone using curly braces and semicolons in Python indicates that they have very little familiarity with the language. Someone forgetting a semicolon in C++ is not a big deal. However, the most frequent issue is for a candidate to get distracted by trying to remember something specific that has no bearing on the result of the interview. It’s usually fine to let a candidate make up library functions that are “close” to what the real library function is if their memory fails them—but equally, inventing “magical” library functions that solve too much of the problem is not a good indicator. In most cases, if someone would use a library in real life, it’s wise to let them use it in an interview.*

When Are Coding Questions Necessary?

Some have questioned the necessity of intense coding interviews and suggest placing a greater focus on design questions, behavioral interviews, and other tests of knowledge.

On the one hand, coding questions in artificial environments under the pressure of an interviewer’s watchful eye can make people perform poorly, making for ineffective coding assessments. On the other hand, coding questions are meant to screen for necessary technical skills. There are people who simply cannot write code or who have only studied computer science and never really practiced engineering in work environments. Looking at experience alone isn’t always a good indicator of how the person will perform in the new role. A coding interview can help ensure that this person is able to do the work.

Another approach is to only ask relatively easy coding questions to verify that the candidate can code fluently, and then use other interview techniques to test other skills. Other resources like past work samples may be reasonable alternatives for measuring the person’s likelihood of success in the role.

Strictly no-code interviews may make sense for certain senior roles, where communication and high-level problem solving are more central to success, and actual coding is more limited. For some junior positions, it may be easier to teach code than it is to teach good communication, so if you’re planning on having the person learn a lot on the job, factors like problem-solving and collaboration skills might be more important. (There are training programs and coaches available to improve employees’ coding abilities in these situations, if there are not enough available resources to mentor them in-house).

​story​ “The best no-code interviews I’ve seen involved diving into a candidate’s resume and talking about past projects—reaching the edge of their knowledge on the topics and probing one step further. It allows them to show off what they know and gives an accurate picture of what they don’t. It requires a very senior engineer to do it correctly, but it works beautifully.” —Laurie Barth, Staff Engineer, Gatsby

​story​ “Dropbox has a pretty high bar for coding questions. It’s not entirely clear if, from first principles, you ought to design a hiring system this way, but it’s the system we understand and operate.” —Alex Allain, Engineering Director, Dropbox