Dear Artemis: Are case studies still an effective way to evaluate candidates?

Welcome back to Dear Artemis: Where you ask us tough questions and we use our collective 45+ years of knowledge in the recruiting space to answer them.

Dear Artemis,

Are case studies still an effective way to evaluate candidates? We’ve always used them, but recently we’ve been receiving some negative feedback and are seeing more drop-off.

- Director of Talent, FinTech Scaleup

Dear Director of Talent,

Great question, we’re hearing this sentiment more and more recently. Case studies often miss the mark as an effective recruitment tool, especially with senior leadership hires.

Why?

Case studies can create ‘false positives’ when assessing a candidate’s suitability for a particular role. Worse yet, they’re often not an accurate representation of the work a leader will do day to day. There are more direct ways to assess for things like research and persuasion - more on this later.

Candidate experience

Interview processes are already lengthy, time-consuming, and often emotionally exhausting. Adding a case study to that process could risk losing candidates by having them opt out. While some hiring teams may think that’s a good thing - viewing self-selection opt-out as a way to “weed through” candidates - we believe it’s not. You may lose out on fantastic candidates who have demanding full-time jobs or other personal priorities that they choose to commit to fully - which is exactly what you want in a candidate, anyway.

Sometimes, candidates worry that organizations use case studies for “free work”. The common workaround for this is creating a generic case study assignment that isn’t directly related to your business. However, this renders the exercise to be an even more ambiguous tool for assessing candidate fit at your company.

Equity and Inclusion

Case studies can act as a systemic barrier to diverse candidates. Any case study/take-home assignment unintentionally favours candidates who have more time to work outside of regular work hours. But that privilege should have very little bearing on their suitability as a candidate. For example: A working single parent may be more qualified than a non-caregiving candidate, but may not have as much time as the non-caregiver. This unequal playing field will negatively impact your ability to select the right candidate.

Illusion of control

Ok, so perhaps you’re thinking: What if we take extra care to ensure that the interview assignment is actually relevant to the job, a positive experience for the candidate, and an equitable process. You’ve time-boxd the assignment, provided compensation to the candidate, established uniform assessment criteria, etc.

Despite all of that - you can’t control how the candidate approaches the assignment if it’s conducted at home. You can’t validate how much time they really spent or who else provided input. Maybe they consulted an expert in their network or dedicated 2x the time you allocated for the work. You’ll be comparing apples to oranges, but won’t even know it.

So, what’s the solution?

We suggest ditching the at-home assignment altogether. Consider a more dynamic in-person activity instead, such as a whiteboard brainstorm, a design thinking challenge, or any other type of activity that you may actually do with your team on a regular workday. You’ll witness (live!) how candidates think through problems rather than just the end result. In most cases, that’s more valuable anyway.

In-person assessments are also an opportunity to invite other team members into the process to get early insights into team chemistry and how different styles may complement or contrast each other. You’ll have direct visibility into the process and be more thoughtful of the candidate’s life outside working hours.

Not sure where to start with in-person assessments? Consider this:

• Start with the Why: Determine exactly which skills, knowledge, and behaviours you are trying to evaluate, and then work backward to create an assessment that can draw this out from your candidates.

• Focus on problems at your company that have already been solved to see how they may have approached things differently, without risking the ‘free work’ red flag

• Inform candidates of the full interview process right from the beginning, including the assessment

• Clearly indicate what you’re looking for from this activity/assessment: What’s the end goal?

• Make sure there’s ample time to debrief, so they can explain their thought process and answer follow-up questions

• Reserve the assessment step for the top 2-3 candidates

• Make it ‘open book’; give the candidates the opportunity to ask questions throughout the activity without being penalized. This is a more realistic representation of what the real role will look like compared to a ‘closed book’ challenge.

We hope this is helpful. We’ve helped many great companies work through refining the executive evaluation process - if you have more questions, feel free to email us at info@artemiscanada.com.

Until next time,

Tyler & the Artemis team