Reviewing a Developer Skill Assesment Tool

Yeah Yeah, Let Us Get To It

TLDR; sure, just go ahead and try out Qualified. It's got an email like UI, lets you see a listing of candidates, assessments, send an invite to someone, track everyone's comments about a candidate, provides an IDE for the candidate to write and submit coding solutions, has a list of thoughtful coding problems for candidates to solve, provides a good UI to review the code and candidate. So yeah...that's pretty much it. I like that it does all those things, it feels feature complete to me. But there's another level to the problem and if you're interested in hearing what I think about that, read on. Otherwise, thanks for stopping by. Good luck.

The Long(ish) Story

I started this post thinking I'd write a review about Qualified. I went back and forth with my opinion on it. As a matter of fact, at Zeal, we've spent about 2 weeks talking about skill assessment tools and interviewing people in general. Partly because we've been interviewing a lot, and partly because I feel like we're thoughtful people and want to genuinely identify and understand the problem.

Kijana Woodard

  1. I invited many of you to take Qualified.io for a spin.

    The idea is that we can use it in our hiring process to bring a bit more structure into the mix.

    I sent everyone the "basic pre-screen" questions.

    They have many languages and levels.

  2. My question to everyone is

    1. What would you personally think about being asked to take such a screening?
    2. Do you think such a tool would be helpful for Zeal?
    3. If so, in what ways would it be helpful?
    4. If not, what's a better alternative?

Trying to ascertain someone's skills is tough huh? I mean, even with a tool like Qualified, it's not really enough. You're still gonna have to spend time thinking about how the candidate solved the problems, what their code looked like, formatting and such; AND THEN, how all of those things map to skills. I mean, exactly what are the skills required to write quality software? How do you even define QUALITY? (if you're thinking it's bug count, this isn't the article for you; come back in 5 years when you're able to recognize the core problems)

Joey Guerra

  1. First things first, what's the definition of quality software? Gotta have a clear answer for that before anything else.

    Every time I think about what my answer to this question would be, I ask my self another question, and that is, "from who's perspective?"

    It seems to me that quality means different things to different people. But if we hone in on who the stake holders are in software, I'd say it's 2, well 3, but I think the business at large is obviously a stake holder, but 2 are really important, the user and the developer (I know, I'm biased, humor me).

    Having articulated that for myself, NOW I feel like I can come up with a really good definition of quality. Here it is:

    As a user of the software, quality is the standard of page load, user error count, and consistency as measured against the previous version’s page load, user error count and consistency.

    As a developer of the software, quality is the standard of readability, discoverability and leverage of existing code as measured against the previous version’s readability, discoverability and leverage of existing code.

    The key points being that it's a standard and measured relative to how it performed previously.

    My definition of quality software

Luckily, I've spent some time thinking about this, so I feel like I can form some opinions. But not everybody has; frankly, there's skill involved in identifying someone's code that results from a particular skill. For instance, good formating is an indication that someone has the skill to write clean code. Hmm. I wonder how much code you'd have to read to be able to make that assessment reliably? I'm guessing more than just a few solutions to a generic Computer Science problem. Not that it's impossible, just that it's not easy.

This was what I was thinking about while using Qualified's code assessment UI. I was a little suprised that I could see a difference in skills when reviewing candidates (they were 2 other Zeal developers :) solutions to short problems. But I digress. I was thinking that Zeal has a bunch of developers, we can just create a code repo and define a problem ourselves for the candidate to solve; maybe we don't really need a tool to do that for us? I dunno yet. If you use Qualified, you wouldn't have to spend money solving that problem. Heh. That seems like a benefit.

And then sometimes someone comes into Slack and drops a grenade that makes us all dive for rabbit holes, figuratively speaking of course.

Kijana Woodard

  1. Great. Now I want to hire Austin Walters

Kenneth Cassell

  1. Yeah I know right! I had the same thought when I was reading it. “I wish we could hire this guy” :joy:

Thanks alot Austin Walters. Somehow you've reached out from the internet, forced Kenneth to read your article and post it on Slack; and making a point that really resonates with us too. People are complex, not only do we have many different skills, but we also have variations of them too, which makes us super interesting and also super hard to put in an Enterprise Job Description box. Crap. This is really hard.

On one side of the spectrum, you have people who are looking for good talent...ah, the "it's so hard to find good talent" fallacy. Alas. Such a nuanced topic. What talent are you actually looking for? I think we can all agree, nobody's born with the skill for writing quality software. It's developed. So what does it take? If someone's really typing out lots of code per day, or quick to solve logic problems, does that mean they'd directly add value to the business? Or better yet, to the teams ability to deliver quality software? And there it is. There's the thing that's difficult to do. Identifying the skills required to CONTRIBUTE to our objectives, while at the same time, identifying the skills that would HINDER us from achieving our objectives (we don't want to hire those people, sorry, not sorry). Qualified isn't going to give you that. You have to figure that out on your own. And you're gonna have to figure it out regardless if you have a skill assessment tool. But I digress. There's the one side of the spectrum that's having a hard time finding good talent. And then there's the other side of the spectrum that isn't. What side are you on? Why do you think some people are having a hard time, while others aren't?

I suspect it comes down to the individuals definition of talent and their understanding of the objectives; and whether or not they spend time to articulate the skills and talent they think will help them achieve those objectives. I personaly look for things like curiosity, grit, evidence of learning, before I think about whether they can solve a logic problem in code. But I think that's because I feel like I can teach someone to write quality software better than I can teach someone to be self motivated, respectful of others, or have a strong work ethic.

Phew. I hope this adds to your Machine Learning Data Model in your head with regards to whether or not to use Qualified. Hiring people is a large problem domain and finding useful tools to help tackle it seems valuable. Hit me up on Twitter (@ijoeyguerra) with your thoughts. I'd love to see what you think.