Online Challenges

You’re reading an excerpt of The Holloway Guide to Technical Recruiting and Hiring, a book by Osman (Ozzie) Osman and over 45 other contributors. It is the most authoritative resource on growing software engineering teams effectively, written by and for hiring managers, recruiters, interviewers, and candidates. Purchase the book to support the author and the ad-free Holloway reading experience. You get instant digital access, over 800 links and references, commentary and future updates, and a high-quality PDF download.

Online challenges (or coding challenges) are tests that a company sends to prospective candidates in order to screen them for specific criteria early in the hiring process, usually before phone screens or more in-depth interviews. Online challenges typically involve a small set of predetermined programming problems. Either the company or a third-party company will have created them, and the hiring team will expect candidates to complete them in a specified amount of time and without any assistance.

When candidate volume is high, screening mechanisms like online challenges can make sense; they are easier for companies to scale and assess than other types of evaluations, and they are less biased than traditional screening mechanisms like resumes. Hiring teams use online challenges to identify unqualified leads.

Companies can learn a lot about candidates by giving them a homework assignment in which they have to write code to solve a problem. These technical screens are almost always scored automatically—once the candidate writes and submits code, the assessment system runs it against a bunch of tests to verify that the code outputs are correct and that the code runs efficiently.

​story​ “At Dropbox, we would go to a campus and would get a thousand people who would apply from that school for an internship, but we’d only have five interviewers on the campus. We would have them use Hackerrank X (a code-screening software) to tackle a simple problem, because what we didn’t want to do was destigmatize the really good people in the pool from doing it—it was basically, ‘can you code or not?’ We were trying to differentiate between classroom students and engineers.” —Scott Woody, former Director of Engineering, Dropbox

Examples of widely used assessment tools include HackerRank, Codility, CodeSignal, and Qualified.io.

Because these assessments take minimal time and effort for companies to give out, they’re an appealing approach to making hiring a bit more fair. The idea is that if someone can pass a coding challenge, regardless of how they look on paper, they should get to advance to an interview. In some cases, online assessments can replace reliance on traditionally poor-signal screening formats like resumes: there is less bias in testing someone’s code output than in placing worth on where they went to school.

Still, there is value asymmetry in online challenges.

Value asymmetry is a state of affairs in which one party derives greater value from a transaction than the other. Value asymmetry in hiring and recruiting can arise when companies require candidates to expend significant effort in the interviewing process, such as asking a candidate to complete a lengthy coding challenge that provides little benefit to them. Value asymmetry can lead to frustration, disaffection, and candidates dropping out of the hiring funnel.

If your company is a household name, you can get away with having value asymmetry in your process because candidates value the chance to work for you so highly. But there are very few companies that are household names; and until yours is one, it’s wise to weigh whether it makes sense to ask candidates to do something up front without giving them anything in return.

Figure: Value Asymmetry as a Function of Brand Strength

Source: Aline Lerner

To combat value asymmetry, companies often choose to send coding assessments just to candidates that they perceive as less marketable; that is, those that don’t look good on paper. One pitfall is that even when those candidates do well in these assessments, they may not get to move to the next step—either because they made a mistake on the online assignment or because after they pass, a recruiter may revisit the candidate’s resume and notice a practical reason that disqualifies them (like location).

​caution​ It’s wise not to make the common mistake of assigning the same online coding challenges for all developer roles. Typically (1) frontend or fullstack, (2) data science or machine learning, and (3) devops roles all are different enough they should be assessed with different challenges. Effective hiring teams test for and assess candidates’ ability to fulfill the requirements of the specific role all along the funnel.

​story​ “Online challenges are incredibly irritating. They’re things like, ‘Create a map of 8 by 14, and then write a program that navigates that in as little moves as possible.’ They’re asking you to build out chess boards for web developer positions. I’m a great programmer, but I’m not a computer scientist. I fail these things repeatedly. I know the people who can pass these kinds of challenges, and they’re the worst programmers I’ve ever worked with. For every one company that uses these, there are four or five that don’t—and I’ve worked for them instead.” —Aaron Saray, engineer

Many companies with high applicant volume at the top of the funnel use some kind of online challenge. They are automated, skills-based, and less biased than using resumes. If you choose the right platform, such challenges can increase hiring efficiency. But they will never give you a complete picture of a candidate’s ability to do a job, and high potential for false negatives remains, especially if you make the mistake of assigning challenges that have little or nothing to do with the skills required for the role. These are screens meant to disqualify poor fits, and are not useful for qualifying further.