Technical Interview Formats

You’re reading an excerpt of The Holloway Guide to Technical Recruiting and Hiring, a book by Osman (Ozzie) Osman and over 45 other contributors. It is the most authoritative resource on growing software engineering teams effectively, written by and for hiring managers, recruiters, interviewers, and candidates. Purchase the book to support the author and the ad-free Holloway reading experience. You get instant digital access, over 800 links and references, commentary and future updates, and a high-quality PDF download.

Selecting Interview Formats

Technical interviews can be conducted in person or remotely, with one interviewer or a small group, synchronously or asynchronously, on a whiteboard, a laptop, or as a take-home test. Which technical interview formats will be most effective—and how those interviews should be conducted—depends on what signals the company finds most useful to gather. Each format collects different information, and each has pros and cons, pitfalls, and associated time and effort investment from both sides. When choosing a format, the company should consider whether it can be scaled and still conducted in a way that is fair to and useful for the candidates.

Deciding on formats can be a balancing act between time savings for the company and interviewers, quality of the signals that are truly predictive, and the need for a positive candidate experience.

Many processes will involve a combination of several formats for technical interviews:

• Phone screens or video call screens.

• Take-home tasks or assignment.

• One or more onsite visits with in-person meetings, one-on-one interviews, pair or panel interviews, or working sessions. Onsites can be technical or nontechnical, or both.

• Fully remote meetings.

Generally, formats move from screening (which help disqualify poor fits) to assessments (to qualify and measure candidate-company fit). Assessments include the technical or nontechnical. As a candidate advances through the interview process, the interview can ask more theoretical questions. Along the way, companies sell to candidates—but efforts to sell usually increase as a candidate moves through the pipeline, all the way to the onsite. Effort on both sides increases the further a candidate gets. But be wary of adding too much to the interview loop in the hopes of further defining the signal. There is a point of diminishing returns, and strong candidates are unlikely to stay through a never-ending process. If you aren’t getting the signal you need, it’s time to look carefully at each format you’ve chosen and figure out what’s not working.

Table: Onsite and Remote Formats

Table: Types of Interviews by Format

Figure: Interview Format Usage

Based on data from their experience with over 900 engineers, TripleByte found that interviewing practices and formats vary quite a bit across companies.

Source: TripleByte

​controversy​The “best” interview format for coding questions is a notoriously controversial topic. While many companies rely primarily on onsite, face-to-face coding interviews, a significant fraction of engineers consider “whiteboard coding” to be intimidating, stressful, and not predictive of job performance. On the other hand, alternatives like take-home tasks have drawbacks as well.

So how do you decide which style to use? Given the trade-offs, the answer may differ by company, by role, and sometimes even by candidate. For example, an internship-style arrangement might work for an entry-level candidate without a lot of work experience, while a laptop-based interview may help a more experienced engineer showcase their skills. Some companies combine methods (for instance, having a whiteboard interview as well as a laptop-based one). You might also offer candidates a choice of which assessment method they prefer, though that significantly increases the company’s work to develop, maintain, and evaluate the different assessment methods in a fair way.

​important​When choosing or evaluating formats, consider how each format relates to general hiring and interviewing principles.

According to Scott Woody, Dropbox used to do a lot of take-homes, because they found they got some of the strongest signal possible on a candidate’s ability to solve problems in unique ways. But they had to start moving away from them because they couldn’t contain the cheating.* If we apply this back to the principles, we see how it was easy (if disappointing) to make the decision to switch formats—it’s not fair because some people are cheating, and it’s not effective because you can’t assess people’s work properly or comparatively. The company had gone about their commitment to the format in the right way: they built up a robust process around take-homes, wrote and used detailed rubrics, conducted blind assessments to minimize bias, and despite all of this, encountered an unavoidable pitfall. They wisely pivoted.

In reality, no technique is perfect, and the bulk of ineffectiveness in the industry results from poor adherence to principles in both design and execution. A well-designed and properly conducted whiteboard interview is more effective than a sloppy laptop interview or a take-home exercise that frustrates a candidate. Select whatever format will cover the skills that correspond to the requirements of the job. This will protect efficiency, effectiveness, and the candidate experience. When in doubt, you may wish to revisit both the hiring principles (of candidate experience, accurate assessment, fairness, and efficiency) as well as the interviewing principles (of minimizing bias, being predictive of on-the-job work, avoiding questions that can be gamed, and being mindful of both your and your candidates’ time).

Technical Phone Screens

A technical phone screen is an early conversation between a candidate and a current employee of a hiring company with technical ability. Most commonly, technical phone screens are conducted by engineers at the hiring company, but they may also be conducted by others, such as recruiters with deep technical knowledge. The content of a technical phone screen can range from a simple conversation to live coding.

​confusion​ Note that for some candidates—particularly high-value candidates, referrals, and outbound candidates—interviewers may begin with a more general first conversation to get to know the candidate and gauge their interest. Depending on their experience, these candidates may skip technical phone screens altogether.

​important​In general, technical phone screens are better for disqualifying obviously poor candidates than for qualifying them. Secondarily, they can help identify potential superstars whom interviewers will want to focus additional energy on courting.

It is easier and more important in a short, abbreviated phone screen to figure out that a candidate doesn’t understand something like recursion and thus fails to qualify than to assess whether they have deep algorithmic ability, which is hard to measure quickly and reliably. Harder questions tend to make bad phone-screen questions because the phone format makes hard questions more difficult for candidates to succeed at, and because even strong candidates may miss any given interview question. Starting the process with harder questions also means you are exposing your question set to a larger pool of candidates, increasing the likelihood of leaks. Therefore, a phone screen process focused on sorting out bad fits will work better than one that tries to search only for stars.

​story​ “At Dropbox, we tried moving hard questions to the front on the theory that harder questions are a better filter and that we would save time downstream. Many fewer candidates made it through the initial screening process, reducing the number of onsites. Only the truly outstanding candidates made it through, and we filtered out a lot of the lower-skilled candidates. However, we also filtered out a large number of candidates who could have been good. On examination, we realized that if you gave a good candidate two hard questions, they would usually get at least one down well and struggle a bit on the other. These people would do well on a full interview panel, but if you happened to give them the wrong question in the the phone screen, you would reject them, which increased the rate of false negatives. We moved to a system where the phone screen is focused on filtering out obvious wrong fits while giving room for truly exceptional candidates to shine.” —Scott Woody, former Director of Engineering, Dropbox

​story​ “At Triplebyte, we get to see how the same candidate does at different companies. And it’s far more common than you might think for someone to be screened out top of funnel at one company and then go on to do really well at another company. This is fine, as long as the screening question that was used really is in an area that the company cares about. There is far more variance in candidate knowledge (even experienced, skilled candidate knowledge) than most hiring managers think. So, make sure if you ask a screening that requires knowledge of, say, time complexity, that you really do need people to understand that. Because there are productive, experienced, senior engineers who don’t understand complexity analysis. And if you don’t require web dev experience, make sure you don’t ask a question that requires someone to understand fullstack dev.” —Ammon Bartram, co-founder and CEO, Triplebyte

Skipping the Phone Screen

The phone screen is almost always necessary because online challenges, whether or not they are combined with resume screens, will not give you enough signal to move someone to an onsite. You can often skip the initial phone screens for candidates whose previous experiences make it clear that they would be minimally capable of performing in the onsite process (for example, that they possess baseline coding fluency and problem-solving skills). Having candidates share open-source work on GitHub can be a strong early sign of programming ability. Phone screens might also be skipped for people who are high-valued or are referrals. If the chance that someone would fail the phone screen is very low, and you have the resources to bring them onsite, it’s wise to do so. The more stages you include in the process, the more likely some number of candidates will drop out.

Removing a step is a simple way to accelerate the pipeline and should not compromise signal if the same material is covered in later stages.

Onsite vs. Remote

Typically, the large part of the interview process happens onsite, but this is changing as more companies build remote teams or wish to accommodate candidates who are not local. You can use many interview formats either in person or remotely; this includes even live coding challenges. A company’s resources and the candidate’s availability usually determine whether a particular part of the process will be conducted remotely or onsite.

Remote interviews have a few specific use cases:

• Typically (but by no means universally), remote interviews are concentrated earlier in the funnel and may include:

  • Online challenges and take-homes.

  • Phone screens, both general and technical.

  • Technical screening questions, questions where the candidate can code in the browser during a screen share, and light behavioral questions. Note that while you may encounter binary decision points or spikes in the signal during remote interviews, this is not a useful format for making more fine-grained decisions.

• You can conduct assessments of candidate portfolios or previous work remotely and asynchronously, but portfolio reviews can also be an onsite activity where the candidate takes the interviewer through a work sample.

• Remote post-onsite follow-up interviews can be used to clarify anything that came up during the onsite and to talk about next steps.

The hiring team will likely build into the pipeline some kind of remote assessment, from technical screens to post-onsite follow-ups, so it’s important to understand the possible constraints as well as the benefits.

Confidence in what signal you’re extracting will be weaker over the phone, so it may be preferable to ask questions with clearer signals, like questions with a right or wrong answer. It’s also important to ensure that every candidate, to the extent possible, goes through the same loop when it comes to remote vs. onsite.

​story​ “The signal isn’t weaker or stronger with remote interviews, it just manifests differently. If I talk to some people onsite and others remote, I’m going to perceive the phone signal to be weaker, because humans just have stronger attachment to in-person experiences and can gather signal from a variety of different cues in person.” —Scott Woody, former Director of Engineering, Dropbox

​confusion​ If you’re hiring for a remote position, should candidates still be brought in for an onsite? Onsite interviews are enormously important for selling candidates on the role, team, and company. If the position is remote or remote optional, have everyone go through the same process, because you’re comparing people, not remote performance compared to onsite performance. If a candidate cannot come onsite because of family obligations, a current job, or other concerns,* put your best, most senior interviewers on that person.

Onsite Interviews

The onsite interview is a series of interviews held at the company’s office for several hours to a full day. Onsites can be a crucial part of the interview experience for candidates and companies alike because they offer an extended opportunity for assessment on both sides. Onsites typically include pair interviews, pair programming, live coding, debugging, non-coding technical interviews, and nontechnical interviews.

​controversy​ Some companies have a multi-day process where the candidate comes back onsite several times. This can slow down your hiring process and lead to a bad candidate experience; but for early-stage startups or key roles, the opportunity to spend more time with the candidate may be worth the risk.

The pros of onsites are numerous—they provide the highest signal you can get. The human factor is important here: candidates get a real sense of the day-to-day output and environment of a company, and if you train your interviewers well, the onsite allows for a more conducive environment for comfortable communication. Onsites are very important for selling the job to candidates, who will get to experience the team firsthand, usually for the first time. Most candidates would prefer to interview in person, though in some cases it is essential to offer a remote option.

Some interviews provide much higher signal in person than they would over the phone. Technical non-coding formats like architectural interviews are nearly impossible to conduct when you don’t have a shared whiteboard to work from. Behavioral interviews are easier to assess when you can also read a person’s body language and facial expressions.

​danger​ A risk of onsites is that they can be a source of bias; they require more evaluative efforts from interviewers than things like coding challenges do, and interviewers tend to feel more comfortable rendering verdicts based on false signal. For example, handwriting or communication style can color how an interviewer interprets code on a whiteboard. But few people are actually qualified to make such judgements. Interviewers might think, “This person was nervous at the whiteboard,” and infer that the candidate is inexperienced or has a reason to be nervous. Such judgments are usually wrong and tend to be based on unconscious biases. Interviewers tend to uprank nervous-seeming candidates who look like them or have a similar background, while penalizing or downgrading candidates for nervousness when they are underrepresented or have a different background. It is crucial to train interviewers and use structured interviews, to keep candidate performance from being a function of who is conducting the interview or what mood they are in at the time.

Structuring the Visit

​caution​ Many believe coding-intense interviews provide the noisiest signal in onsite interviews. Ideally, an interview loop includes multiple sessions and multiple technical formats to avoid a situation where one bad interview tanks a candidate. In general, coding questions can also be the most tiring for candidates because they are being asked to solve new problems rather than talk about previous work. Discussing previous work gives you some of the best signal on a candidate’s coding ability and their problem-solving skills. On the other hand, if you have too many questions or formats, you will wear candidates down. It’s important to measure out the day, with easier, more straightforward interviews coming earlier. Coverage of other technical skills (such as deep diving into a prior project) and nontechnical interviews can come later in the day.

​story​ “To get in more questions (better signal) without tiring the candidate, design the questions so that they flow together. For example, you might ask them to first build a simple API. Then add rate-limiting. Then maybe solve an algorithm problem that comes from efficiently rate-limiting across multiple machines. Then maybe add a frontend client. The thematic connection of the problems allows the candidate to stay focused on one codebase. This is not super common across the industry, but it’s something that we’ve seen work really well for us.” —Ammon Bartram, co-founder and CEO, Triplebyte

Lara Hogan provides a template you can use to determine the list of signals you need to gather to make a hiring decision, and assign those signals to people in different interview slots. This is a great place to start planning out who will cover what for your onsite interviews. Marco Rogers adds even more detail to his post that includes an onsite interview schedule template, notably including a second interviewer in each session and capping the total number of sessions at four.

Finally, having some kind of wrap-up at the end of the day with a hiring manager is an important part of giving the candidate some closure on the day and a chance to ask questions and hopefully meet their future manager. Additionally, given how important the manager-team member relationship is, this is an important part of building toward a successful close. You can also learn a lot about a candidate’s perspective through the questions they ask, and while you will be giving them time during the other interviews to ask questions, they may have new questions and insights by the end of the day and may be feeling more comfortable.

Typical One-on-One Interview Structure

Technical one-on-one interviews often follow a simple form. This general overview illustrates how most in-person interviews are conducted:

Pre-interview prep. Most interviewers will require at least a few minutes before the interview to get into the right mindset, make sure they understand what questions to ask, and what the expectations for the candidate are. This need not take more than 15 minutes, but new interviewers may experience nervousness leading up to the interview throughout the day.

Greeting. Welcome the candidate and set them at ease—they may be experiencing high stress, information overload, or energy drain. A positive greeting gives them a chance to settle into the conversation and get used to the interviewer. Energy and enthusiasm are infectious; interviewers who bring genuine excitement to the interview can help a tired candidate show what they are capable of. Ask if they need to use a bathroom or would like something to drink. It’s the human thing to do. Making them comfortable is not only the right thing to do, it will also make it easier for you to assess their actual capabilities.

Interviewer introduction. It’s both helpful and just friendly for interviewers to let the candidates know who they are before diving into the details. It’s awkward to talk to someone you know nothing about, and intros build some initial rapport while providing a chance to highlight one or two things the interviewer likes about the company.

​danger​Some interviewers might try to make conversation without realizing that it is against the law to ask certain questions in interview settings. Common illegal interview questions include asking about marital status or age.

Introductory questions, usually short and easy. Most interviews start with an easier warmup question or questions, which could be a simpler version of a difficult question that will be asked later. The warmup allows the interviewer to adjust their approach based on whether things are going better or worse than expected. Some candidates won’t realize the initial question is not the hard part of the interview and will over-solve or over-think it. Good interviewers help the candidate understand the flow of the interview so they can manage the time and nudge the candidate away from being unnecessarily thorough or careful in this phase.

You might start with a version of the question where a simple solution would work. For example, you may pose a coding question where the first version of the problem can be solved with storing everything in memory.

Follow-up questions, usually meatier. A gradual increase in difficulty ensures that candidates can be assessed on a spectrum of how far they were able to get, not whether they had a single “aha” moment or got stuck on something far too hard. The meatier follow-up question or questions should consume half or more of the overall interview time.

Using the example above, once the candidate understands the problem and has something working, you can introduce new requirements that up the challenge, like increasing the amount of data to a degree where it will no longer fit in memory.

Questions from the candidate. The final question-and-answer section allows candidates to learn more about the company and provides insight into what the candidate is concerned or excited about. It’s important for interviewers to write these down for use in the closing process.

Tell them what will happen next. This is the time to clarify the next steps in the process, including the approximate timeline. Helping the candidate estimate when they will know if they’re moving forward can help reduce stress, leading to a better candidate experience. For instance, for phone screens or after the last session in an onsite, it’s important to let the candidate know that there will be a later follow-up and when they can expect to hear back. In between onsite interviews, the candidate should know when and where they can meet the next interviewer—whether they should wait in their current interview room or whether they will be escorted to another room.

​danger​It’s important for the interviewer to refrain from sharing their assessment with the candidate during or immediately after the interview. Even when the interviewer feels certain of the outcome, the interview is just one data point in the process, and their opinion might change when they see the full set of feedback from colleagues or take some time to reflect. It’s helpful to remember that the period during and immediately after an interview can be stressful for candidates, and that stress and adrenaline may cause them to be less likely to properly interpret (or handle) any feedback. Thus, an appropriately neutral but generally positive closing is to thank the candidate for their time and indicate that you enjoyed the discussion.

​important​It is, however, equally important to deliver a response in a timely manner. If you take too long to give a candidate any feedback at all, they may read the silence as impolite disinterest on the part of the company or convince themselves they don’t want the job anymore.

In the late stages of the process, candidates often ask about the actual solution to a coding problem they were presented with or whether there was a better approach than the one they took. Time permitting, answering neutral technical questions like this is fine and can lead to a more engaged technical discussion that gives the interviewer more signal—a candidate that is truly interested in better ways to solve a problem is usually a more thoughtful engineer.

Pair Interview

In pair interviewing, two interviewers simultaneously work with and assess a single candidate. Typically, one interviewer leads the interview while the other observes. Microsoft’s Developer Division and Twitter, among others, use pair interviewing exclusively.

​confusion​ Some companies or teams use a form of pair interviewing to onboard new interviewers; this is called shadowing.

Add another colleague into the mix as a second interviewer. I’ve found that it not only brings a higher fidelity and signal to the conversation, but also breaks down the flow and nature of the conversation. Sometimes a person will be talking to the candidate, and other times, they’ll address their colleague. I find questions and answers bouncing between three points opens up the discussion and evaluation much more.Marco Rogers, veteran engineering manager*

Pair interviewing can yield more complete signal from an interview—if one interviewer notices something that the other interviewer doesn’t, they can collaborate to assess what the outcome should be. They may help reduce bias, by bringing two perspectives to the same situation. This format can also help to root out bias on the candidate side.

​story​ “When pair interviewing with a male colleague, I’ve sometimes seen candidates only speak to or make eye contact with him. This is behavior that likely wouldn’t have been seen were I interviewing solo—it would be super unlikely for a candidate to ignore the one person talking to them. Pair interviews can help to reveal if a candidate treats one member of the pair noticeably differently, which can be useful signal for how they might treat different teammates.” —Ryn Daniels, Senior Software Engineer, HashiCorp

Pair interviewing also creates quality related feedback loops, since interviewers can share their thoughts with one another on how the interview was conducted. That said, pair interviewing effectively doubles the amount of interviewer time spent per candidate, and it is wise to evaluate whether it is worth that extra cost.

Pair interviewing does add extra interviewer load, but interviewers seem to like it since it gives them increased confidence in their assessment and a mechanism for learning and feedback—a bit like paired programming.Jan Chong, Senior Director of Engineering, Twitter*

The likelihood of success in pair interviews may depend on the specific pairing. For example, having a pair made up of a manager and a more junior person can be problematic. Risk is lower if the junior person is shadowing the manager and therefore is not expected to question the candidate directly. But if they’re meant to be partners in the interview, the junior person may feel intimidated by the manager and be unwilling to risk asking the wrong questions of the candidate. Structured interviewing and careful planning can mitigate this risk; each person in the pair can be assigned a tack to take, or an area to focus on, or specific questions.

A pair of developers who are of equal rank and similar disposition may work best; variant personality style causes greater risk. When a candidate is focusing hard on answering questions, having to switch between, say, a low-energy interviewer and a high-energy interviewer can be extremely difficult and tiring. This does not set the candidate up for success.

​important​ It might seem intimidating to a candidate when more than one interviewer walks into the room—you can mitigate this by letting the candidate know in advance that you will be conducting a pair interview.

Panel Interviews and Presentations

A panel interview is one in which three or more interviewers simultaneously evaluate a candidate.* Typically, companies select interviewers to represent a variety of perspectives regarding the open position.

Panel interviews have downsides for the candidate experience. Having to address multiple interviewers at once can feel like an interrogation; introverts in particular usually do better in one-on-one settings. One upside is that panel interviews give the candidate a chance to meet more of the team. But for most people, panel interviews are intimidating.

That said, there are a few possible benefits of this format for the company.

One company benefit is that in a very consensus-based hiring process, a panel interview makes it possible for more people to be involved. It may also be useful when multiple hiring managers are looking for different types of candidates or are trying to route a candidate to the right role or team. However, with everyone asking different questions, the likelihood that the interview will get deep enough for a strong signal is low.

Panel interviews can be beneficial for reducing bias, when answers can be gauged by multiple people of varying backgrounds. However, it’s important not to misrepresent the diversity of your company or overburden underrepresented employees.

​caution​ Although it may seem that panel interviews can give signal on how a candidate will work in a group, panels may not reflect the team the candidate would work with or may lack the dynamic of a collaborative team, particularly if the group feels artificially put together, or no one has specific questions to ask that they couldn’t get signal on in another format. Inadequately trained panelists may give off a vibe that they do not want to be there, hurting the candidate experience and downgrading their perception of the company. It’s hard enough for candidates to adapt between different types of people at the same time, especially in a situation that is already stressful. So if you decide to use this format, it helps to make sure each panelist has a clearly defined reason for being there.

In the end, most people just hate presenting in front of groups. Panel interviews often do not meet the candidate experience principle.

A similar, and often preferable, alternative is to have the candidate give a presentation to the team. One hiring manager* told us that a team he worked for tried presentations and ended up scrapping it because the signal they got was of so little value. One hire they made based on the person’s presentation ability ended up not being a strong fit for the role; they had gotten signal on the wrong kind of communication ability, one that wasn’t relevant to collaborative work. While engineers do sometimes present ideas to groups, like in design reviews, it’s a relatively infrequent activity and usually less formal than a typical presentation.

On the other hand, presentations are fairly common for research or scientist positions, because the work requires conference presentations. Similarly, if the role you’re hiring for requires similar work as a core component—product managers or a sales position—a presentation evaluation makes more sense.

Whiteboard Interviews

A whiteboard interview is a technical problem-solving assessment that takes place in real time and typically involves a candidate writing code and sometimes diagrams on a whiteboard while onsite.* A similar kind of interview can be done in person with pen and paper.

​controversy​ Whiteboard coding questions are controversial in the industry and their merits are hotly debated.* Whiteboard coding questions do not accurately mimic the environment that a candidate will usually work in—they have no access to a compiler, editor, or reference material as they normally would. They also bias toward candidates with fresh, detailed memory of a language or library and against developers who prefer a more iterative model of writing and testing code incrementally. Reacting to these potential flaws, Hiring Without Whiteboards provides a popular, extensive, collaboratively sourced list of companies that don’t use whiteboard coding.

​important​ Looking at whiteboarding more broadly, however, can dissolve what appear to be inherent constraints of the format. As more than one senior engineer put it to us: “No one is good at coding on a whiteboard.” Accepting that can be the start of something interesting. If hands-on coding tells you what an engineer knows and how they use their knowledge, whiteboarding interviews may better cover the why—why does the engineer make each choice? When conducted well, whiteboarding gives signal on an engineer’s communication and collaborative abilities, as well as how they think through problems and theorize solutions.

​story​ “The whiteboard is just the visualization mechanism and is useful in any talking interview. The whiteboard could be a laptop. Generally the whiteboard is just a way of drawing things. It’s a visual mode of communication.” —Scott Woody, former Director of Engineering, Dropbox

Physically writing a lot on a whiteboard can be unfamiliar or stressful for those who’ve not done it much. A helpful alternative is to offer a pencil and pad of paper to a candidate who is not comfortable on the whiteboard. This requires that the interviewer be able to sit next to them to review and discuss as they write.

Hands-on Coding Interviews

Hands-on coding interviews include all formats where a candidate uses a computer and not a whiteboard. They are often part of an onsite, but can also be conducted remotely.

In a hands-on coding interview, an interviewer will give the candidate a technical problem; the candidate will typically use a code editor to solve it. This provides signal on whether a person has done this kind of work before, the nature of their coding style and problem-solving style, and what practical set of skills they have. Because most programmers are more comfortable with writing code than completing a task on a whiteboard, hands-on interviews move along more quickly, allowing you to go deeper and get more volume of signal.

Pros and Cons of Hands-on Coding Interviews

Are hands-on coding interviews better than whiteboard interviews? It’s complicated!

Some advantages of hands-on coding interviews include:

• Technical candidates are more familiar with the coding experience, which may be particularly relevant for candidates who rely heavily on the features of their IDE.

• Hands-on coding interviews are based on a more natural style of writing and debugging code incrementally, which allows the candidate to display confidence in their work.

• The candidate can use reference materials.

• The candidate can provide an exact transcript of the code at the end of the interview.

• Because hands-on interviews are live, they offer little room for cheating.

The primary downsides of this approach are:

• Not all candidates have laptops, so the company may need to provide them.

• Laptops may have hardware or software issues that delay the interview; whiteboards rarely do.

• Unless the company provides scaffolding, the candidate may need to write a bunch of boilerplate code to get started. If the company does provide scaffolding, it likely needs to do so for a large number of languages, and the logistics of getting the scaffolding to the candidate can present a hurdle. (One possible alternative is to have the candidate modify or debug an existing codebase, allowing the collection of additional signal on other skills relevant to software engineering like reading and working with new code.)

• The candidate may get caught up in such non-essential elements of the problem as making sure their code compiles or looking up the exact right library function, rather than focusing on solving the more fundamental aspects of the problem. While these are also skills required of an engineer, they’re more basic than the translation of ideas to code, and detract from evaluating that.

• Most programmers do not perform their work in front of other people; the pressure of having someone watch them code can be enough to throw candidates off their game.

• It can be somewhat awkward for the interviewer to watch the candidate work without staring over their shoulder. While this can be mitigated by using an app like Coderpad, the editing experience when using such an app will not be what the candidate is used to.

Those downsides notwithstanding, for the right kind of question, having people code on a laptop can offer a better candidate experience. Hands-on, synchronous work more closely mirrors a real working environment, with back-and-forth between the parties, and with no expectation that a candidate do something perfectly without feedback.

While online challenges screen for basic coding skills, synchronous, hands-on formats are closer to assessments, because they allow you to measure how a candidate reasons their way through problems. This results in better signal on a candidate’s talent, not just their skill or knowledge base. Interviewers can course correct, and can increase the difficulty of the question or questions as the interview proceeds, depending on how the candidate is performing. Signal is also more accurate because candidates can’t ask friends or look up answers.

Compared with online coding challenges, hands-on coding interviews are more expensive; they require an interviewer to be present in person, over the phone, or in a shared screen environment, as well as the cost of transporting the candidate to an onsite. They also require far more training and preparation from interviewers.

Compared with the whiteboard environment, where everyone is expected to fumble (or to hesitate or be more playful in how they use the whiteboard), candidates are expected to be more precise and efficient when working in a code editor. Some fumbling might be expected for people earlier in their career, who may perhaps be not as familiar with different coding environments or who haven’t spent thousands of hours coding—especially when working with an interviewer looking over their shoulder. That doesn’t mean a less senior person isn’t right for the role you’re hiring for. If the person can be trained on the job, how much they fumble in hands-on work doesn’t matter as much; the interviewer’s focus then becomes evaluating talent over skill.

Some companies opt to constrain candidates on language or environment, putting candidates in a place they may not be familiar with. Language requirements are normal, but environmental requirements are rarely standard unless you use a specific coding environment like iOS or Android. Barring role requirements, restricting language or environment is an artificial constraint that will give you poor signal on their ability and may cause you to lose a third to half of your candidates.

On the other hand, unconstrained choice on language and environment means interviewers must prepare more thoroughly and may receive more variable signal. Has a candidate made a smart language choice, like using a typed vs. untyped language? If they use a typed language, errors will be apparent; if they use an untyped language, the interviewer needs to know how to find the errors the candidate might make.

To remedy this, it’s best to train interviewers to ask the candidate doing the coding to talk them through their work, rather than just watching the candidate deliver the code. “If anything, more talk, less code,” as Scott Woody put it to us. This can be harder than it seems, especially when the interviewers are junior engineers who want to appear accomplished and impressive to candidates. It’s very reasonable for an interviewer to say, “I haven’t seen a solution like this before. Can you talk me through it?”

​story​ “I tend to recommend that interviewers try to take an attitude of ‘How would I interact with someone I was pairing with or mentoring?’ This cuts down on seemingly adversarial attitudes that can make candidates nervous. It’s hard to be at your best when it feels like someone is wanting or waiting for you to fail.” —Ryn Daniels, Senior Software Engineer, HashiCorp

​story​ “It’s gauche to judge someone purely on language expertise. You can learn a language if you’re a good engineer. That said, if they need to be an expert in Go, go ahead, force them to use Go in all their interviews. That’s appropriate. What I find more often is an implicit expectation that you’re an expert in whatever language you use. In reality, we’re all searching the internet for help with our work. That’s just more realistic.” —Scott Woody, former Director of Engineering, Dropbox

In short, in the best case, hands-on coding can give you fine-grained, practical signal. What do they know, and how do they use that knowledge? As with other formats, building clear rubrics and communicating the signal you’re trying to get from a hands-on interview can help avoid judgments based on work style or approach.

​candidate​ We suggest Yangshun Tay’s Tech Interview Handbook Cheatsheet, especially when preparing for hands-on interviews—it includes how candidates can explain their work in a hands-on. There are also helpful reminders here for preparing for other types of interviews, like phone screens.

Some Hands-on Interview Formats

A live coding challenge is an exercise or work sample that provides the company with content for assessment that mimics the work a candidate might do on the job. Within live coding challenges, there are two variations: a company may ask a candidate to work on a project on their own while onsite, or the candidate may be assessed in a pair programming interview, in which the candidate interacts directly with the existing team on a project.

Pair programming and live coding make the most sense when the role is for a coding-only position. With senior positions, which require a lot more than coding ability, these formats might be a waste of time—there are other ways to evaluate how someone thinks through code and approaches problems, and how they lead others.

The challenge is to design something close to what the candidate would be doing on the job; giving a puzzle on recursive algorithms when the role will not require that kind of work will not give you relevant signal. It also has to be possible to complete the challenge in a reasonable amount of time.

With pair programming and live coding, candidates can’t cheat. The skills demonstrated are practical. The candidate may have told you that they know what a singleton pattern is, but can they write one or recognize one in code? Live work helps to track how the candidate applies their knowledge, which a candidate can help with by walking the interviewer through what they’re doing in detail. This may be especially important for recent graduates.

Live coding presents the same kinds of challenges as other hands-on formats—most engineers do not spend their time coding in front of others. The pressure and awkwardness of doing so can be challenging enough to throw off a candidate and give you false signal on their abilities. You’ll see how quickly and efficiently the candidate produces work, but only to a point; the pressure of the live environment can hobble some engineers. Live work is a skill in and of itself. Dealing with that anxiety or not being affected by it measures something different from coding ability.

​story​ “An interview is as contrived as a blind date. When you meet someone by accident, glorious music plays and you get to see the real them. A lifetime movie is made. When you go on a blind date or a Bumble mixer, it’s so predatory and awkward. Same thing with live coding. If you start failing, you’re going to spiral out of control because of the stress. In real life, someone corrects you, a bug is filed, you go home, you come back the next day and you fix it. With the coding experience live, you make a mistake, EVERYONE knows it, and you can’t reset all that well unless you’re very controlled.” —Aaron Saray, engineer

​story​ “I’ve seen companies with open-source efforts tackle a ticket with the applicant. It’s interesting because it requires the interviewer to find a ticket before each interview that is appropriate, and if the project isn’t sufficiently large, that’s hard to do for multiple candidates; the ticket will always be different. But it gives two options: either they can pair on the open ticket or, if it’s a senior candidate, they can review an existing PR and make adjustments.” —Laurie Barth, Staff Engineer, Gatsby

Debugging interviews are exercises in which candidates are asked to find solutions to real-world bug problems in codebases. Debugging is a critical task for most engineering roles, and an interview of this type can give insight into how the candidate reasons and works with their tools. These interviews also test for how well a candidate can read others’ code and make changes that respect existing designs.

Debugging may give a narrower slice of insight into the candidate than a take-home, which usually entails a mix of debugging and building.

The challenges of debugging interviews are mostly logistical. Debugging interviews require a lot of time on the part of the interviewers to develop the right question and build up scaffolding code in a language and environment with which the candidate is relatively familiar.

Note that some debugging interviews may be “great in theory, bad in practice.” As many as a third or even half of your candidates just won’t be able to solve the challenge. The more standardized language and development of iOS or Android environments make them a possible exception.

Take-homes

A take-home assignment (take-home or takehome) is a coding task given to technical candidates to complete on their own time. Candidates are typically given a day to several days to complete a take-home.

​controversy​ Take-homes are controversial. While there are many pros for the companies assigning them, they are less valuable in terms of the candidate experience. Nonetheless, they do have some advantages for candidates.

Benefits of Take-homes

​candidate​Take-homes remove a lot of the stress associated with onsite challenges. Candidates get to use their own tools and work in the style they would if they were on the job. They can review and iterate on their work, take time away to think or rest, and rewrite. One senior engineer put it this way: “Most employees ‘take home’ their work if you think about it. You get work, you go away and think, you do it, you sleep, you come back and review it. That’s how our jobs work.”

For companies, take-homes have arguably the lowest false negative rate of any interview format—“the truest signal,” as Scott Woody, former Director of Engineering at Dropbox, put it to Holloway. A few factors account for this:

• Take-homes give the candidate enough time to do the work in an environment that they’re comfortable with, so you eliminate the noise of a whiteboard interview.

• It’s very hard to hide weaknesses in coding ability in a take-home, and follow-up conversations can tell you almost everything you need to know about how a candidate thinks through problems.

• Follow-ups also allow you to weed out and correct for any negative signal or false signal—for example, if a candidate cheats, talking through their work will help you figure that out.

Follow-ups are an important part of the take-home evaluation; in this respect, take-homes are the first step of a larger conversation. A good take-home will mimic assignments the candidate might reasonably be asked to do on the job and will give you practical signal on their abilities, creativity, and style.

Here’s how a sample assignment might progress. Let’s say the take-home is something like “Build a simple web-based calculator app” or “Build an AI version of Tetris.” When the candidate returns, you might read the code and interrogate it together, bringing the initial asynchronous assessment into a synchronous evaluation:

1. You ask the candidate to critique their work.

2. You may then ask them what they would do given another 20 hours; or say, “How would you invest 10 more hours on this?” Questions like “What feature would you remove?” and “What feature would you add?” will further refine your understanding of their work.

3. An additional useful question is, “What shortcuts did you take, and why did you think that was the right shortcut?”

Downsides of Take-homes

​caution​ The major downside of take-homes is the time commitment they require. Because the market is so competitive, asking a strong candidate to give up their weekend for a take-home can lead them to drop out of your process. Senior candidates may feel that being asked to do a take-home is a waste of their time. For these reasons, take-homes usually make sense further along in the funnel. They require a lot of engineering time and investment from interviewers and candidates, so they aren’t ideal for screening. Some companies opt to pay candidates for the time they spend on these assignments, but this doesn’t always make a difference. A senior manager at Dropbox told us that before the company pivoted from take-homes, 20% of candidates would simply not complete them. Less-competitive candidates were more likely to complete the assignment, because they didn’t have competing offers. The pass-through rate was close to 10%. If you’re asking candidates to invest 15 hours, and their chance of passing through is 10%, the value asymmetry is strong.

Despite the high signal achieved, interviewers, too, spend hours of their time designing take-homes, scaffolding in multiple languages, and reviewing code, and with such a low pass-through rate, this hardly pays out.

​danger​ The other major con of take-homes is that they explicitly discriminate against people who have families or adverse financial situations, or who work more than one job. If these candidates are in the hiring pipeline at more than one company, they may receive multiple take-home assignments at once, making them impossible to complete, and this may cause the candidates to drop out of your process.

Take-homes also open up the possibility that a candidate may cheat by asking friends to help or collaborate. Additionally, with a take-home, you’re asking candidates to work in a vacuum, which doesn’t match to most work environments, where you’re hopefully able to ask questions and get feedback as you progress.

​caution​ Take-homes are difficult to timebox and thus difficult to assess fairly. If candidates have been given 48 hours to turn in an assignment, you might be comparing candidates who spent 40 hours with those who spent only 2.

One way to avoid this false signal is to pay people an hourly rate for the assignment; but candidates can lie, either saying they worked more hours so as to receive the pay or fewer hours because they want to look impressive.

There are situations in which the pros of take-homes outweigh the cons. Smaller companies may find it easier to assign take-homes than to expend the time and resource investment in a longer pipeline, where multiple interviews would be needed to get the same signal. Younger engineers trying to break into the industry may prefer take-homes because they provide a chance to demonstrate skills they haven’t yet had a chance to prove on the market. If there’s a candidate you haven’t gotten clear signal from yet, adding a take-home to their pipeline will usually tell you one way or another whether the bet will pay off. Scott Woody, former Director of Engineering at Dropbox, told us that people who tend to shine on take-homes have nontraditional backgrounds: “They’re hackers, or they never took CS in college, and they’d fail out of our normal process. But we can see they’ve been doing all this practical work on the side, so let’s give them this practical thing and they’re going to build something singular.”

Take-home Tips

Using a tool like Takehome.io can help with timeboxing take-homes. It might seem like the option of timeboxing would help solve a lot of the cons of this format, but many engineers hold the opinion that time limits introduce further artificiality that compromises what could otherwise be a clear signal. There are tradeoffs any way you approach it.

If you do choose to give take-homes, it’s important to be clear with the candidate that the results will only be used for evaluation and not to produce work for the company. You might also provide an upper bound on the amount of time a candidate should spend on the take-home. When sending candidates the assignment, it’s important to let them know what it is that you will be evaluating—the code? the creativity? the speed? This will help ensure that they don’t waste time on something that won’t translate as much to the assessment and the eventual job. You likely also will want to avoid noting things as nice-to-haves unless they are truly necessary for the assessment.

One idea worth noting to help make the candidate experience better is to replace the take-home with a project that is done in the office during the onsite. Such a project still requires a logistical burden, but has the benefit of feeling like a symmetric exchange of time, particularly if it replaces multiple interview questions. The goal is to mimic the benefits the take-home has for candidates—let them work alone.

​candidate​ Understanding the reasons why a company might choose a take-home problem can help candidates prepare. This guide from Jane Phillips has a host of practical suggestions for tackling take-home coding challenges, along with an FAQ on common scenarios, like needing more time or what to do if you’re not familiar with a language or framework in the take-home problem.

Prior Work Assessment

Some companies choose to ask candidates for past work samples rather than asking them to write code (though you can do both). The nice thing about this approach is that it allows you to see something that the candidate actually did in a real-world setting. However, it can be difficult for many candidates to provide this kind of work sample if they don’t have an open-source presence, and evaluating these work samples may take more time and require a great deal of interviewer effort to evaluate. Prior work assessments can be:

• Synchronous. The candidate walks the interviewer through a completed project or portfolio.

• Asynchronous. The candidate sends work to the interviewer for them to review, and/or the interviewer reviews the candidate’s open-source projects (likely on GitHub).

• Both asynchronous and synchronous. The interviewer looks at the sample without the candidate present, then meets with the candidate to discuss the work.

Looking at past work can show you the technologies with which a candidate is familiar and how they architect solutions. Past work that is open-source or a “passion project” demonstrates what the candidate is really interested in doing and allows them to shine. When not explicitly associated with a previous position, this kind of work also can show you where candidates made mistakes or curious choices and where they may have cut corners. As with discussing a take-home, looking at past work can offer illuminating discussion with the candidate: Why did they make those choices? What would they have done differently, given the time or hindsight?

Unfortunately, it’s not always easy to tell exactly what work is the candidate’s and what was done collaboratively or primarily by a colleague, or when they came into the project with their own contribution.* Candidates might share work that isn’t relevant to the job requirements, and looking through work that has nothing to do with the job requirements may be a waste of time.

If interviewers do choose to dive into specific previous work, it’s best to give the candidate advance warning and also a general direction of the kinds of topics you would like to cover. This allows the candidate time to refresh their memory of the work they choose to showcase. At the same time, building in extra time puts some pressure on the interviewer to actually dive deep to ensure the candidate is not just giving a rehearsed presentation.