editione1.0.8
Updated August 24, 2022You’re reading an excerpt of The Holloway Guide to Technical Recruiting and Hiring, a book by Osman (Ozzie) Osman and over 45 other contributors. It is the most authoritative resource on growing software engineering teams effectively, written by and for hiring managers, recruiters, interviewers, and candidates. Purchase the book to support the author and the ad-free Holloway reading experience. You get instant digital access, over 800 links and references, commentary and future updates, and a high-quality PDF download.
Ideally, interviewers will record their feedback on the candidate as soon as possible. The fresher the interview is in the interviewer’s mind, the more complete and objective it is likely to be. Additionally, since next steps rely on this information, waiting a while to record your feedback can slow down decision-making.
The write-up justifies the decision with concrete evidence based on the rubric, by identifying which parts of the rubric were or weren’t met. A sample write-up based on the technical question above might look like this, for a performance evaluation of “fair”:
The candidate struggled with this problem overall, earning no more than a “fair” on the rubric and a “no hire” on the interview. Mapping how they did to the rubric:
The candidate asked appropriate clarifying questions (“What kind of data is stored in the tree?”, “Do you care about performance?”); they assumed the output would be printing to stdout—the interview was off to a good start.
The candidate struggled to write a correct version of the program; their initial attempt only printed the left and right branches of the tree one level deep.
The candidate struggled to identify their bugs when they walked through the program themself; for example, they failed to recognize that they didn’t handle the empty tree correctly and had to be prompted to handle multiple levels deep. With significant, repeated hinting (including giving a concrete example case) the candidate eventually did fix their issues.
The candidate wrote reasonably idiomatic language.
The candidate couldn’t properly describe the Big O of their solution (they claimed it was O(1) instead of O(n)).
dangerIt’s important that no interviewer be exposed to other interviewers’ feedback before they have submitted their own. Seeing other feedback can bias an interviewer, which can diminish the quality and fairness of decisions. This is especially true if someone more junior is exposed to the opinion or feedback of someone more senior. Check to see if your ATS (if you’re using one) can help hide other interviewer’s feedback as it gets submitted.*
The level of detail required in interview feedback can vary from company to company, depending on how that feedback will be used. For example, feedback used as notes to help an interviewer remember key points to discuss during a debrief session may be less detailed than feedback that will be shared with and used by an independent hiring committee.
That said, the more comprehensive the feedback, the less room for relying on memory that might fade (even over the course of a few hours or days) or “gut” instinct (that may be prone to bias). And the hiring team may want to revisit that feedback further in the future—for example, when deciding whether to reconsider a candidate or to analyze the hiring decision. Providing a structured form for interviewers to provide feedback makes this easier. Many tools can help enforce timely and structured submission of feedback.
This section was written by Kevin Morrill.
Comprehensive feedback will do the following:
Paint a narrative. An interviewer’s feedback ideally will clearly convey what they talked about with the candidate. What questions did the interviewer ask, and how did the candidate answer them? Are there code snippets that can be included in the feedback? This helps the team draw conclusions from the collective set of interviews. One of the best ways to make a narrative clear is to quote the candidate, instead of jumping straight into your own interpretation.
Tie positive and negative aspects to core competencies. Anything that went well or poorly in the interview will ultimately relate back to the competencies, skills, and traits you are interviewing for. At the very least, it’s helpful for the interviewer to connect observations outside core competencies to a predicted situation on the job.
Make a predictive statement about on-the-job performance. Why will a particular behavior the candidate demonstrated matter on the job?
Be written down. All the data collection in the world is useless unless you write it down and have it available for the debrief session and hiring manager or committee.
Convey a clear decision. Interviewers hopefully will walk away from the interview with a clear sense of whether the candidate is a fit. If they’re not convinced, that’s a default “no hire”—there is no such thing as a neutral opinion. The spectrum of “hires” extends from “strong no” to “strong yes.” For a “strong yes,” you will feel like you need to chase them to the parking lot and get them on board immediately, and you would be worried if they joined a competitor. Below that, but still not in the “no hire” range, you may feel like they will raise the average level of performance on the team. Some companies allow the interviewer to specify how confident they are of their rating.
Convey secondary signals. For instance, maybe the candidate performed well in a technical interview, but was somewhat rude. It’s useful to disentangle those two signals.
Identify gaps. Comparing across interviewers’ feedback, the team can also figure out where there are areas or open questions that need follow-up (either via another interview or through conversations with the hiring manager or recruiter).
Support an audit. if you ever make a hiring mistake and have to fire someone, one of the first postmortem activities is to evaluate the interview loop and figure out what happened. If you have great notes, you can learn and improve the process.
Here’s an example of an interviewer’s effective written feedback:
Recommendation: HIRE
Pros: High intelligence (keen awareness of concretes, able to employ abstractions), results oriented, good communication, aptitude for organizing code effectively.
Cons: Questionable organization and time management skills.
Details:
Matt worked out to be a very impressive candidate. I am convinced he would do very well with the kinds of problems we solve and immediately drive value to our customers.
I started by talking with Matt about evolving our architecture to support data on locations coming from multiple sources and then making a verdict on what the actual location is based on all of that data (rather than our “first-in wins” model). He probed on the problem of when conflicts arise, doing the math to realize that even a 1% conflict rate is about 2,300 results to manually review. He assessed that this is too much human work and that an algorithmic approach is in order. I like that he carried through the math and actually thought about it at a concrete level. Weaker candidates just take the problem as given and plow forward assuming it must be a problem worth solving.
He talked about technology choice by saying “Do you want to stay Postgres?” I clarified that we’re on MySQL and said he had the power to change technology but needed to have a good reason to do it. He talked through what that would mean and immediately gravitated to indexing needs as being key. For key/value-oriented storage, it would be harder to efficiently query on things like last_updated, as you’d only have one key to work with. At one point in his thought process he said aloud, “My intuition says…” and proceeded to explain his thinking. I took this as a very good sign, since Courtland brought up that he charged forward on a weak answer to interest rate impact, even though he probably didn’t know what he was talking about. This kind of verbal queue says to me that when he’s working in his comfort zone, he documents his assumptions and critically evaluates them.
At one point I foolishly said that he should probably just ignore the overwritten data; I didn’t mean to trick him, but in retrospect this worked out to be a good example of challenging a manager when they’re wrong. He made a good case for how you could use changes in the data from the same source to tell you the accuracy of a location. This was also another good example of him thinking intelligently on the fly. Love that he could take something abstract, like whether to save data, and pull it down to concrete about a specific startup’s migration to make the decision.
We then talked through what the guts of his black-box “Rules Engine” would be. I’ve actually been thinking a fair amount about this, having coached Bryan Chang on his dev design. He proceeded to lay out a design while thinking on his feet; it was far better than my own thought process that benefited from far more time and knowledge about our challenges. Right out of the gate with no prompting/leading he said, “If statements would be stupid” you want “rules and authority rank.” He came up with a system where we could pass around references-to-function that embodied the rules. Each function could return either a final answer or a weighted answer. I had never thought about using weighting, which is a pretty cool thought.
To close out, I asked him the five-minute communication question. He seemed to get right away what things he should be preparing for. He said aloud, “Let me make sure I can encapsulate key points.” He asked if I knew anything about Shakespeare (for some reason many candidates that are worse communicators don’t take the liberty to ask, and this shows in their work as they tend to be very black box and harder to work with). He wrote an outline before he started. He then proceeded to give an impassioned explanation of Shakespeare’s use of verse. The big red flag is that he went over by at least 50% in time and didn’t seem phased by this despite all his careful preparation. In some cases I’ve seen this be a predictor of poor organization that carries through into delayed projects on the job.
danger Be careful discussing candidates on a public forum like Slack. Even if the audience is limited, if one person makes the wrong comment, Slack discussions can easily spiral in unhelpful, inappropriate, or unfair directions. Without care, interactions like this could even become the basis for a lawsuit.
By contrast, a senior employee can create a clearly organized, in-person debrief that avoids these challenges. However, remote teams may treat Slack discussions as the equivalent of in-person debriefs. If this is the case, you can train interviewers on appropriate conduct in this forum, and managers can make sure they steer the conversation if it becomes unhelpful.
While the interview process includes the whole interviewing team—and good decisions here require transparency—there is no reason to reveal information about candidate performance beyond the decision-making audience. It is important to make sure candidates do not accidentally see their own interview feedback or discussions about them, if they do eventually join the team.