editione1.0.8
Updated August 24, 2022You’re reading an excerpt of The Holloway Guide to Technical Recruiting and Hiring, a book by Osman (Ozzie) Osman and over 45 other contributors. It is the most authoritative resource on growing software engineering teams effectively, written by and for hiring managers, recruiters, interviewers, and candidates. Purchase the book to support the author and the ad-free Holloway reading experience. You get instant digital access, over 800 links and references, commentary and future updates, and a high-quality PDF download.
Because engineering does not take place in isolation, everyone you hire must be able to work well with other people. As your organization grows, and as individuals become more senior, the nontechnical aspects of the job become increasingly important.
There are several ways to evaluate nontechnical skills, including behavioral questions, situational questions, and role-playing. It is critical to anchor your assessment of nontechnical skills in your defined company values, so as to ensure you are hiring people with the skills that map to the expectations you will have for them once they join. Just Googling “icebreaker questions” or asking candidates to “tell me about your hobbies” will not produce results that will serve you well. It’s important that nontechnical interviews be just as serious and structured as every other aspect of the interview process.
As discussed in Preparing Interviewers, it may make the most sense to have senior people conduct nontechnical interviews, which are typically harder to assess than straight coding interviews. Some managers recommend conducting all nontechnical interviews as pair interviews, including one senior person and one less senior person. This can help to reduce bias when assessing subjective answers and helps to train junior interviewers in a kind of active shadowing. (Recall that you may have to adjust for the possible pitfalls of having different-level pairs.)
While many companies wait to conduct the nontechnical interview until the onsite after technical screens and assessments, some managers opt to incorporate a kind of nontechnical screen earlier on.
story “Incorporate the most important two or three behavioral questions into the prescreen, to give nontechnical skills equal weight to coding ability. Our current practice is that we have a 30-minute phone prescreen (or Zoom, face-to-face, whatever is lowest barrier for the candidate). Then we do back-to-back technical and behavioral-fit interviews, each an hour long.” —Benjamin Reitzammer, freelance CTO
Behavioral questions assess nontechnical skills and values alignment by asking candidates about prior experiences and the way in which they handled specific situations. Many behavioral questions aim to extract specific positive or negative examples of the candidate’s performance against the expectations of the organization.
story “To make this anchoring easier, it helped our team to spell out the company values with as specific as possible descriptions of how someone who exhibits the company values behaves or acts, such as: ‘You openly communicate your thoughts, feelings, and concerns and contribute to an environment which allows this.’ Of course this can also be done by formulating negative behaviors and actions that will not be tolerated. Based on these concrete descriptions, it’s possible to derive situations in which they may arise and then formulate behavioral or situational questions based on these situations. For example, an organization that values feedback-seeking individuals might use the above description for their value of ‘feedback-seeking.’ A possible behavioral question then could be, ‘Tell us about the last time you provided feedback to team members.’” —Benjamin Reitzammer, freelance CTO
Examples of behavioral questions:
“Tell me about a time you had to deal with a difficult co-worker.”
“Tell me about a time you made a mistake. How did you handle it?”
“Give me an example of a goal you didn’t meet and how you handled it.”
“Tell me about a time a project you were responsible for was falling behind. What did you do?”
“Tell me about a time you had to deliver bad news. How did you handle it?”
“Tell me about a time you mentored someone.” Possible follow-up questions include:
“What did you learn from it?”
“What would you do differently next time?”
“What did you think worked particularly well?”
“Give an example of a project where you had to convince your peers or boss to take an approach that you suggested, but they were hesitant to try.”
“Tell me about a conflict you had with a co-worker.”
“Have you ever taken guidance from someone more junior than yourself?”
“Tell me about a time you had to take an unpopular position.”
“Name some improvements or experiments you made in your most recent position.”
“What was the most helpful feedback you received from a colleague or manager?“
“Tell me about the last time you provided feedback to team members.”
“Tell me about a time you gave a compliment to a team member.”
“Tell me about a situation where a first solution wasn’t the right one. How did you find out and how did you iterate on the solution to find the right one?”
caution As with non-coding technical questions, assessing a candidate’s performance in these situations is difficult and requires judgment and experience. There are several common patterns to watch for that can be a negative signal in these interviews:
Some candidates will consistently blame others for bad results and not take ownership or acknowledge their own failures. This is generally a concerning pattern of behavior—a red flag.
Candidates may avoid getting into specific details. This is often a sign that they’re uncomfortable doing so and may indicate that they realize they mishandled a situation and that getting into detail would reflect poorly on them. They may also be nonspecific about what they personally did vs. what other people did, and this may reflect situations where the candidate contributed less than it appears.
Candidates may use “I” rather than “we” when describing the work done by their team. This might be a sign that a candidate is not a team player. However, be careful in judging a candidate harshly for using the first person; they are in an interview to assess their performance, and they’ve likely been asked to focus on the results of their work. The choice of “I” vs. “we” may simply come down to speaking style. This is a place to give some benefit of the doubt until you see a strong pattern backed by additional evidence beyond speaking style.
Finally, new interviewers will often be too trusting of the candidate’s self-assessment about something in their past going well. Instead, interviewers can be digging deep to look for base facts as evidence and can draw their own conclusions about the candidate’s work. It’s a good idea to practice the art of the follow-up to dig deeper into a candidate’s answers.
Behavioral questions have the benefit of focusing on what a candidate actually did in a particular situation. But because there might be candidates who have never experienced the situation you describe, having a hypothetical version of the question can be a great idea.
Situational questions (or hypothetical questions) ask a candidate to explain how they would approach a specific challenge. The advantages of situational questions matched to job-relevant skills are that they allow an interviewer to evaluate candidates against a more objective rubric than behavioral interviews, and they challenge candidates to make choices in real time.
important Well-designed situational questions avoid scenarios where it’s relatively easy to identify what should be done but not so easy to take the appropriate action when presented with a difficult situation. For example, it’s easy for someone to say that they would own up to a mistake; it’s harder to actually do it. Instead, it’s best to focus on situations where the hard part is identifying the core problem and/or devising a solution. Additionally, incorporating behavioral questions into the interview process will allow you to gauge how the candidate might actually behave.
You can often rephrase a behavioral question as a situational one through a slight change in wording. For example, “How would you deal with a colleague who doesn’t deliver on their commitments?” is the situational form of the behavioral question, “Tell me about a time you worked with a colleague who didn’t deliver on their commitments.” This may make the two types of questions seem interchangeable, but some experts have strong opinions (often backed by data) on which form is more effective at evaluating candidates.
Ultimately, it matters less which you use (a mix is probably fine) and more that your questions are structured and your interviewers are calibrated to give consistent feedback. Adam Grant gives some specific details on situational questions that can help interviewers overcome confirmation bias.
A topgrading interview is a type of behavioral interview in which the interviewer asks targeted questions about a candidate’s experiences in all prior roles, proceeding chronologically through the candidate’s career. If you make one of the questions about each prior role be the name of the candidate’s boss, the company can confirm the candidate’s description of their experiences through reference checking.
To learn more about topgrading, see “The Art of Interviewing 10x Engineers,” a podcast discussion between Dan Portillo and Wade Foster; the “Topgrading Interview” section of Matt Mochary’s The Great CEO Within; and Who: The A Method for Hiring, by Randy Street and Geoff Smart, who helped coin the term.
Role-playing interviews ask candidates to act out specific challenging scenarios, such as interacting with a difficult co-worker. Role-playing interviews give the interviewer a chance to see how a candidate would realistically approach a particular situation.
Role-playing interviews require very well-defined scenarios so that the candidate’s performance can be accurately evaluated. Even with a well-defined scenario, these interviews can be hard to evaluate because of the contextual nature of problem-solving and interpersonal interactions. For example, a candidate who relies on relationships and deep insight into other people may not perform well in a role-playing interview where they’re asked to interact with someone they’ve never met.
It’s beyond the scope of this Guide to develop a bank of nontechnical questions, especially because these questions depend so heavily on your company’s goals, values, and mission—not to mention the specific details of the role for which you’re hiring. We’ve included a few categories of questions you can consider, however, and resources to help you dive deeper into which questions will work best for your needs. New Relic has an excellent post on evaluating potential managers that covers their approach to designing nontechnical questions and their rubrics for evaluating them.
story “You have to let the candidate surprise you. It makes the process of coming up with these questions easier, because you don’t have to come up with all the possible answers. You can look at who are the super productive and thoughtful people on your team, the very valuable people on your team, seniors, lead people, and then ask something specific about them: ‘What do they excel at?’, ‘In what situation did they do something really great?’ Then try and turn that into a question. One example might be that ‘a senior person stood up for someone else in a daily standup meeting.’ Now come up with a related behavioral question. Identify situations where senior people really shone a positive light on the kind of behaviors you’re looking at. On top of that, look at your company’s values and build questions from there. ‘You are the kind of person who values feedback.’ Ok, now you know you can ask, ‘What kind of feedback cycles do you have in your work? How do you give feedback?’ Determine what you optimize for, and build questions from those behaviors.” —Benjamin Retizammer, freelance CTO
When thinking about values alignment, it is critical to know your company and team’s values, and then use your nontechnical interview questions to assess how the candidate has previously demonstrated—or undermined—those values.
important Remember, values alignment is not the same as culture fit. If someone has behaved unethically in the past or demonstrated a lack of integrity, that’s a good reason not to hire them; if someone doesn’t seem fun, that’s not.
Values alignment can often be determined through behavioral or situational interviews that test for how a candidate makes decisions, passes judgments, or how they react to making mistakes. This is not about delivering “correct” answers—the point is to assess whether a candidate’s values related to work style, communication, and mission align with the company’s.
The most effective questions focus on sussing out a candidate’s values and allow you to evaluate them based on how well they line up with company values:
If a company values humility/humbleness. Does the candidate demonstrate ownership for their mistakes or have examples of taking feedback from others?
If a company values biasing for action. Does the candidate have examples of making decisions in situations where the path forward was unclear?
If a company values depth of expertise. Does the candidate have examples of going the extra mile to understand something?
If a company values working as a team. Does the candidate have examples of supporting teammates through a difficult time or putting the team’s goals ahead of their own?
| Prompt | Evaluation |
|---|---|
| “Tell me about when you gave a compliment to a team member.” | How the person feels about giving positive feedback. Do they think about the word appreciation, is it about positive feedback, or very technical things like, “I like your code.” |
| “Imagine you join and everything here has gone wildly successful. Three years from now, what would you want to look back on and be proud of?” | Mission alignment and passion. Are they excited about the core mission? How much do they care about being a great manager and supporting their team? Do they only focus on their own accomplishments? |
| “Name some improvements you’ve made or experiments you’ve tried.” | Self-awareness and what they view as improvement. Again, do they only focus on their own improvement and/or narrow improvements (“I made function X faster”), or do they have a broader view on the value of experimentation and process improvements? |
Another thing to screen for is agency—how much responsibility does the candidate feel comfortable or eager to take on when it comes to solving problems? This is especially important as the company gets bigger.
When assessing agency, it’s important to ask both situational and behavioral questions. Not every candidate will have had the same opportunity to demonstrate agency in their careers, especially if they’ve been in unsupportive work environments—but they should be able to tell you what they would do in a hypothetical situation.
There are questions that can give you signal on how a candidate feels agency that don’t require them to have had a leadership position. For example, a lot of people come to the manager to complain about something at the company. But what you want is someone who will come to you and say, “I saw this problem, and here’s how I can make it better.”
Foundry Group co-founder Jason Mendelson screens for agency with the following question: “Can you share a moment from your career where you felt like you’d been slighted?” Everyone’s got one.
Next, ask them what they did about it. Do they throw their boss under the bus, do they burn their peers or those below them? Do they go higher?
story “I interviewed a salesperson who delivered a great answer to the agency question. He had a shared quota with a salesperson, and she went on maternity leave. So he went and asked the boss, ‘Since she’s on maternity leave, I’m not responsible for the whole quota, right?’ And the boss said, ‘No, you need to hit all of it.’ And he’s like, ‘That’s bullshit.’ And so I asked him, ‘What did you do?’ And he said, ‘I fucking hit the quota.’” —Andy Sparks, co-founder and CEO, Holloway
| Prompt | Evaluation |
|---|---|
| “What got you interested in Acme Inc.?” | General alignment. This rarely totally weeds out a candidate, but sometimes raises red flags where the candidate is interested in totally different things than what the company is focused on. Superstar candidates will also have a chance here to distinguish themselves by the energy they answer with and will often have way more to offer here than average candidates. |
| “Imagine I was a new engineer joining your team at [their current/previous company], what are some key things you’d want me to know right away about the customers of that product and how it shapes the way it should be built?” | Curiosity, bias for understanding customer and product needs. Great engineers won’t just leave it to the product team to figure this out; they’ll actively be curious about what makes their customers tick and be able to communicate it to their team. Weaker candidates here will often just confuse you with a jumble of information about different market segments or platitude type statements. Star candidates help you right then and there to understand the key types of users of the product, what each one needs, and often one or two special details about their psychology that wouldn’t be obvious. |
| “What’s a recent fairly large project you worked on at your last company? Roughly when did the project start and finish? At the halfway point of that timeline, tell me a bit about how you were sizing the project up? What percent complete did you feel you were?” | Risk assessment and ability to tell if a project is really on track or not. Some candidates will only be able to tell you basic stuff here (“Well we finished half the stories in the SCRUM plan, so we must have half way, right?”). Strong candidates understand how to look back at their project plan, but they go much further. They could talk about things like: Is the customer scenario actual coming together? Are we able to show it to people and see it work for them? Are we running into new tech issues we couldn’t foresee that we’ve proactively raised with affected parties like product? They might talk about how they front loaded the riskiest part, and they’re able to qualitatively speak to how they’re seeing it really behave now. |
| Follow-up: “Towards the tail end of that project, what factors were you considering when thinking about whether it was time to ship?” | A standard answer would be something like “the stories are all implemented so we’re done” or “all the tests passed.” Stronger candidates certainly get those things, but they add in more nuance: Is the customer scenario or the key original objective really coming together? They thought about whether they should also add some other element, but realized they needed to just get it out there so they could get feedback. They had a solid plan for deployment and operations. They accounted for how future development experience would work for others (e.g. wiki is updated with how to pull the repo and setup dev boxes). |
One of the most important signals to get from nontechnical interviews is how a candidate communicates. No matter what kind of role you’re hiring for or at what level, communication is key. If an employee struggles to communicate an idea with their peers, boss, or junior engineers, they’re going to be frustrated. They might blame everyone else for not understanding them, or even worse, be disappointed and begin to feel isolated. The vast majority of technical hires need to be able to collaborate and coordinate. To do that, not only do they have to be able to share ideas, thoughts, and challenges—they have to be able to do so quickly.
To assess a candidate’s communication style, Kevin Morrill came up with a high-signal assignment:*
I want you to explain something to me. Pick any topic you want: a hobby you have, a book you’ve read, a project you worked on—anything. You’ll have just five minutes to explain it. At the beginning of the five minutes you shouldn’t assume anything about what I know, and at the end I should understand whatever is most important about this topic. During the five minutes, I might ask you some questions, and you can ask me questions. Take as much time as you want to think it through, and let me know when you want to start.
There is a wide distribution of possible topics here. How to braid hair, how to install drip irrigation, the difference between precision and recall in search quality assessment, the basics of brewing beer, quantum mechanics, the function of reduplication in John Milton’s Paradise Lost, or training a dog to sit.
While what the candidate chooses to explain to you is definitely interesting, the signal you gather is from how they do it. First off, how much time did they take—if any—to come up with a topic and sketch out their five minutes before letting the interviewer know they were ready? Did they write themselves an outline? Did they set something up on their laptop or a whiteboard? Some of this is presentation style, but whatever they do (or don’t do) before they start talking can tell you how much they value organization, planning, structure, and preparation. The bottom line: those who prepare always do better.
The candidate’s delivery style can tell you about their empathy. Do they pause to check if you’re keeping up, asking, “Does that make sense? Are you with me so far?” Are they trying hard to reach the audience where they are, by using analogies or posing questions? If you try to sidetrack them or throw them off a bit, do they say, “That’s not what I meant” or “Perhaps I didn’t explain that well,” and try again? Of course, be careful not to play mind games. Some candidates won’t feel comfortable pushing back on the interviewer.
Another thing to check for is whether the candidate sets themselves a timer. Some people don’t need to because they did debate in high school or have a great internal clock, and others might not have heard the bit about a five-minute limit or choose to ignore it. If a candidate is neither of these, they should set themselves a clock.
controversy Some people who like to use this question are looking for whether the candidate sticks to the five-minute mark, docking them if they go over. Kevin Morrill, the author of this question, says the interviewer shouldn’t stop them at five, so they can see how closely the candidate was listening or how much they overestimate their abilities to self-regulate. Others may be more forgiving of the time limit if a candidate seems excited by their subject or thinks the interviewer wants to hear more. A candidate is likely to expect the interviewer to cut them off when they’re satisfied.
Candidates may stop before the five-minute mark; if they’ve communicated the most important things to you about their topic, they’ve done their job. The purpose of the five-minute communication question is to tell whether, if someone has a great idea about improving the product or speeding up a system, they can communicate it to you in less than five minutes.
Less strong candidates tend to:
jump into a presentation without a plan for what they want to get across
finish either way too fast (so that the interviewer didn’t really learn much) or go way overtime (demonstrating a lack of planning)
not ask questions to learn about the audience
never check in to see if you’re following an explanation
make tons of assumptions about what you already know or understand.
Stronger candidates tend to:
think a bit about what topic makes sense to tackle
come up with an outline for what they want to cover
ask you what you already know about the topic
draw analogies to the things you already know
check in with you: “Does this make sense to you?”
use the whiteboard to express themselves
get to something that’s actually interesting about the topic
give you back a summary of what was covered
finish on time.
Whether someone will be a good manager or not is the subject of countless books, courses, and blog posts (and perhaps a future Holloway Guide!). In lieu of covering this in detail, we provide a few suggested questions to include for assessing management ability and suggest additional recommended reading.
| Prompt | Evaluation |
|---|---|
| “Tell me about your management and leadership philosophy.” | The basic answer here is something like “My job is to make the goals clear, and then get out of the way.” Star candidates know not to be needlessly overbearing, but they have tons more to offer than that. “I work with my team to understand what talents they have and also what motivates them. When someone works on a project, I make sure I know their blind spots they don’t understand, and I always have some mitigation to compensate (e.g. more frequent check-ins, training, get them a mentor, etc.) I make sure that my team gets feedback. I make sure the things that matter are in my team’s control (e.g. great tools, monitoring/instrumentation so they can see what’s going on)” |
| “Have you ever had to fire someone? What led up to it? What did you do when you realized things were off track?” | There are few “wrong” answers here, you’re more looking for the depth of insight the person has accumulated here. Examples of what you might hope to get: Understanding where they could have given feedback sooner; Understanding where the employee wasn’t getting a clear enough message on impact, and how they personally could have done better here; Understanding of how they’d improve hiring process; Understanding of how they’d improve their own feedback process; Understanding of how to go through a performance improvement process. |
| “In our initial conversation, we talked about the immediate needs of the team. Do you have any questions about that? Tell me how you would go after these challenges in the first 90 days?” | The answers to this question will depend heavily on the role—the degree of strategy and proactivity you’re looking for will vary significantly if you’re looking for a development manager or a VP of Engineering. But this person should demonstrate awareness of what is required of their level (or ask good clarifying questions to get there), and provide ideas and tactics accordingly. |
| “When a problem arises on the team, how do you decide at what point to step in and address it?” | Some candidates might not admit to any problems, or a tell a story about when their manager solved a problem for them and how they’d emulate that. Stronger candidates acknowledge that no team is without problems, and has an example of how they constructively resolved a conflict with respect and empathy. |
First Rounds’ 40 favorite interview questions
Workable’s question bank
Devskiller’s list of 45 behavioral questions
Big Interview’s guide to behavioral interview questions
caution Interviewing is an active process that requires the interviewer to be completely engaged. This requires that interviewers not be on their phone or check email or Slack. If you take notes on a laptop, turning off notifications will prevent interruptions. Candidates can tell when the interviewer is distracted, and it makes their performance worse. Likewise, the interviewer’s active engagement at all times will ensure they are collecting the clearest possible signal.
Being engaged with the candidate also ensures the interviewer feels like a human being to the candidate. An interviewer who sits silent and stone faced will intimidate the candidate, who will then underperform.