by Jono
The relationship between technical interviews and real-world performance remains one of the most debated topics in software engineering recruitment. While you may be reading this because of a recent interview experience we shared, this post aims to explore a broader and more important question: how can companies better identify talented developers? Although many organizations have established interview processes they consider sufficient, the challenge of false negatives – rejecting qualified candidates – continues to impact both companies and developers 1. As Joel Spolsky notes, "A lot of good programmers end up getting rejected — while, even worse, companies end up hiring people who are good at passing tests, but under-perform in the real world" 2. Drawing from industry anecdotes 3, professional discussions 4, and academic research 5, this post examines real-world examples and alternatives to traditional coding interviews.
I am a lifelong developer. When I discovered programming in adolescence I took to it and never stopped. After decades of being a full time individual contributor my love of coding only continues to deepen.
I always have my hands in open source and side projects. After the work day is done it is not uncommon for me to put in another days worth of work on my personal and open source projects. Many of these projects also directly benefit my place of work 6 7 8.
Managers say I am enthusiastic and known for going above and beyond to work out innovative solutions. I am eager to learn and like to mentor. I regularly give tech talks, non-tech talks, lead projects, and security workshops 9. See my reviews from peers and managers. They can be verified on linkedin.
I do well on all parts of the interview process, but not on live coding questions. Most professions do not have an equivalent hurdle in their process, but these are common place in tech. Many experienced engineers, particularly at senior levels, view these assessments with skepticism or actively avoid organizations that rely heavily on them 10. They consider it a red flag or they find them insulting.
Still, companies rely heavily on these pass/fail "tests". In my tenure as a programmer I have actually seen interviews improve over the years by relying less on these assessments. But some traditions die hard 11. As technologists, we should strive to keep up with best practices.
Having an hour long tech question tends to rely on landing the "trick" to the answer early on. If you don't happen to know the answer from previously doing the same one or miraculously stumbling upon it in the first few minutes you will be wasting the next 40. Some engineers have a knack for these types of questions - but it is not a good measure of the quality of their work.
Consider a magic trick: initially, it captivates audiences precisely because the solution seems impossibly clever. Yet once revealed, the same trick often appears deceptively simple, leading viewers to wonder why they didn't see through it immediately. This psychological shift reflects perfectly the dynamics of technical interview questions. This phenomenon can be explained by what psychologists call "the curse of knowledge" 12 or the "illusion of transparency" 13 When an interviewer is given the answer to a coding question it becomes obvious to them. They feel it is trivial by virtue of knowing the answer. This is not their fault. It's a natural bias, but one we can address.
The problem compounds when companies reuse interview questions over extended periods - sometimes years. Engineer interviewers naturally refine their understanding of the problem space, discovering edge cases and optimization opportunities. This deepening expertise can widen the gap between their perception of the problem's difficulty and a candidate's first-time experience with it.
Another common pitfall occurs when companies frame interview questions around their domain expertise, even when hiring for general software engineering roles. For instance, a biotech company might ask candidates to implement a gene-pair categorizing algorithm, despite not requiring prior biology knowledge. The employee lives and breaths these questions every day. So does everyone around them. That's likely been what they have been working on for years. Such questions can create an unintended expertise bias, as interviewers who work with these concepts daily may struggle to view the problem from a newcomer's perspective.
In practice these types of questions rarely come up at work. But if they do, we enjoy it. We share the problem with the team and try to figure out the best solutions. Sleep on it. Come back the next day and talk some more. Engineers are problem solvers and love a good challenge, but that experience is different then a timed pass/fail "test".
Much of the same can be said for interview sessions focused on algorithm whiteboarding. Balance a red black tree? When would someone ever need to implement that? In fact, they should not. A good engineer stands on the shoulders of giants and knows to not roll their own cryptography 14.
Here are several alternatives to traditional live coding interviews I have seen or implemented myself at companies that can better assess a candidate's real-world engineering capabilities:
Provide candidates with a substantial coding assignment they can complete in their natural development environment. This approach allows for more complex problems than live coding sessions, as developers can utilize their usual tools and resources – reflecting actual workplace conditions 15. Consider:
- Setting clear time expectations rather than strict limits
- Requesting documentation and test coverage
- Create your own unit tests that can check the answers
- Using platforms like Coderpad or Codility for streamlined administration
- Ensuring the scope remains reasonable and respectful of candidates' time
Present candidates with existing code and evaluate their analytical abilities through various exercises:
- Identifying potential issues and red flags
- Debugging specific problems
- Conducting mock pull request reviews
- Discussing optimization opportunities
- Implementing test coverage strategies
- Analyzing algorithmic complexity
Have candidates present their own project work, which provides insight into their:
- Development process and decision-making
- Technical communication skills
- Understanding of trade-offs and improvements
- Testing strategies and quality assurance approaches
- Project management and prioritization abilities
Share pre-designed algorithms or pseudocode for candidates to implement in code. This method:
- Better mirrors real-world development scenarios
- Focuses on implementation skills rather than problem-solving under pressure
- Creates a collaborative environment for discussion
- Allows evaluation of coding style and testing approaches
Present a variety of shorter, focused technical discussions that might cover:
- Database schema design and optimization
- Algorithmic complexity analysis
- Data structure selection and trade-offs
- Security best practices and PII handling
Consider offering candidates a choice of several of the different assessment methods above. This approach:
- Accommodates diverse working styles and preferences
- Promotes inclusive hiring practices 10
- Allows candidates to demonstrate their strengths
- Respects different time constraints and commitments
These alternative approaches can be implemented within the same time investment as traditional coding interviews, while potentially yielding richer insights into candidates' capabilities. If you want to improve the effectiveness even more, regularly gather candidate feedback and refine your methods over time.
Footnotes
-
Howell, Max (2018). What's the logic behind Google rejecting Max Howell, the author of Homebrew, for not being able to invert a binary tree? Quora ↩
-
Cassel, David (2018). Joel Spolsky on Stack Overflow, Inclusion, and How He Broke IT Recruiting The New Stack ↩
-
Storm, Adam (2020). Why Senior Engineers Hate Coding Interviews Medium ↩
-
Ellis, Ike (2013). I will no do a tech interview Hacker News ↩
-
Behroozi, Mahnaz; Parnin, Chris; Barik, Titus (2019). Hiring is Broken: What Do Developers Say About Technical Interviews? IEEE Symposium on Visual Languages / Human-Centric Computing Languages and Environments ↩
-
jonocodes (2016). Branchr - Feature branch monitoring for product development ↩
-
jonocodes (2017). Envoy- Test framework for microservices using Docker ↩
-
jonocodes (2023). Mystique- Python test helpers for loose data matching ↩
-
OWASP Juice Shop - Insecure web application for security trainings ↩
-
Henry, Bradston (2022). Why I Stopped Interviewing with Companies That Require a Coding Test DEV ↩ ↩2
-
Goodin, Dan (2019). Microsoft says mandatory password changing is “ancient and obsolete” ↩
-
Curse of Knowledge Wikipedia ↩
-
Illusion of Transparency Wikipedia ↩
-
Morrow, Susan (2019). The Dangers of “Rolling Your Own” Encryption Infosec Institute ↩
-
Gentleman, Daniel (2022). Live coding interviews are terrible - and they might be discriminatory LinkedIn ↩