Failed Attempt at Job Simulation

Problem

"In the past I had experience with a company that used job simulation and homework as interview strategies. These tasks are normally designed to give an accurate preview of what the role would entail. Furthermore, they help weed out candidates that aren't a fit for your organization. Unfortunately, these two methods didn't work for us, here's why."

Actions taken

Poor Planning

"The simulations and homework were taking up too much of the candidates' time. At a certain point, people lost interest because they were too long. More so, it was assumed that candidates were interviewing with other companies at the same time, companies who had an easier interviewing process. So we were at a disadvantage because of the poorly planned length of the tests."

Validation on-site

"It became apparent that the interview process on-site was more efficient. We had enough successful hires who didn't go through the job simulation and homework stage that we eventually just phased those stages out. If the on-site process wasn't working so well, maybe we would have put more effort into developing and growing the job simulation and homework areas."

Lessons learned

• "We found that lengthier simulation and homework worked really well for higher demanding roles. So positions that are highly coveted - such as data science or even product management - allowed us to be more selective by purposefully planning for longer tests."
• "I think there is a fine balance between hypothetical testing versus actual work. Even within our organization I have challenged hypothetical questions that has nothing to do with our work. We have been doing this for quite some time, and have never encountered that type of problem nor have any reason to believe it would occur in the future, so why ask it. Don't waste valuable time by asking about things that you will not encounter in your day-to-day work. There are core competencies like data structures and algorithms that should be evaluated but in our case instead of asking tricky/puzzle question we relate it to a task where that skill was useful and have the candidate go through that problem by whiteboarding or actual coding session."
• "For roles which might not require a lot of actual experience we do ask hypothetical questions that relate to your work, even if you don't make the candidate actually follow through to a clear solution. These types of questions give you an idea of what the candidate knows, what they would do, and how they're going to get there versus having them give you an accurate answer or a perfect line of code."