Earlier this year, our EiR David Mytton, wrote about the first step in hiring – how to source candidates. Once you have applications, then you need to evaluate them to decide who you might want to hire.

In this follow-up piece, David shares more about how to think about the interview process. Regardless of how urgent the need is to fill the position, finding the right people, not just for the role today but for how your business will change in the future, is crucial to success. This post will take you through how to create a robust selection process for hiring engineers.

The goals of the process

You have to remember that you are still in a sales process. You are not just trying to match applications against your person spec but you are also trying to convince them to accept the offer you might make at the end. This means there are several goals to consider:

1. Evaluate applications against what you are looking for in team members now, and in the future. You need to balance the requirements of the job today with an ability to adapt as the business changes. This is particularly important in early-stage startups. Past experience may be relevant to demonstrate ability to execute, but knowledge of specific technologies is probably not – the best engineers can learn new skills, languages, frameworks and systems.
2. Continue to demonstrate why your business is a great place to work. This comes in multiple parts, the first of which is well before you even get applications. Building your profile and supporting website materials is important for getting applications in the first place. It is just as important that the interview process runs smoothly, the candidates always know where they are at, what they need to do next and what the timeline is. You need to provide regular updates and fast responses. Their time must be valued more than your own and you need to explain to them why they should be joining the company if you make them an offer. You can never take for granted that just because they have applied to you, they will actually accept any offer.
3. Build a diverse team. This is assisted by the design of the process but also requires you to have the appropriate HR policies in place e.g. flexible working, generous holiday allowances, clear maternity/paternity policies, etc. Thinking about this from the beginning and designing your processes to consider the challenges of diversity means you do not need to do things like positive discrimination, which I do not think is a good way to tackle the diversity problem in tech. The goal is to increase the diversity of the application pool and run an unbiased process to select the best candidates from that pool. Google has some useful guides on diversity in general and there are several good resources for working on gender diversity.

The basic foundation for running a good engineering interview process is valuing the time of the candidates. They likely have full time jobs and/or consulting gigs, so you cannot ask candidates to spend many hours on the phone, doing coding tasks or building projects. Of course they will need to give up some time to dedicate to the process but you should work hard to minimise it.

Step 1: Application

The usual application is a simple form which asks the candidate to submit their basic details, a CV/resume and a short cover letter explaining why they are interested in the job. The cover letter is the most important aspect and the only element that is actually examined at this stage.

In the job and I include an instruction which asks the applicant to mention a keyword in their cover letter. If the keyword isn’t present then the application is instantly rejected. This is specifically to filter out mass, shotgun-type applications and to test for attention to detail.

The best people will usually only ever apply to a small number of positions. You want to find people who take the time to consider the company and role well in advance of ever applying, which means reading the full job ad and description.

Where possible, this step should be automated. Only collecting the minimum amount of information e.g. email, cover letter means you can systematically ignore any other details of the application, such as the CV, name, which might introduce bias. Be aware of protected characteristics and things you cannot ask.

Just like college degrees being mostly irrelevant for engineering positions (unless you have some very specific scientific knowledge you require), some companies are now excluding CV submission entirely. This is worth considering as another way to remove potential for bias. The only thing I find CVs useful for is to research interview questions in advance, but everything you need to know you can simply ask the candidate when you speak to them later.

Step 2: Writing exercise

I have found there is a good correlation between ability to write well and coding ability. Programming is all about clear and accurate communication, whether that is directly in code itself or communication about the project with real people!

I test this by requiring candidates to do a short writing exercise whereby they have an hour to research the answer to a particular question, and write up the response. The question should be relatively easy because the focus is on their written answer. You are simply looking for accurate spelling and grammar. Any mistakes should mean an instant rejection – if they are unable to write such a short piece without mistakes or proper proofreading then that indicates a lack of care and attention.

The task should take no more than an hour and you are not looking for technical accuracy of the response. This is purely an assessment of clear and accurate communication.

Step 3: Coding exercise

Designing a good coding exercise is tricky. It needs to be representative of the kind of skills you need for the role. It should allow the candidate to demonstrate a wide range of skills, from writing clear code to tests and documentation. And it should be straightforward to build in a short period of time – a couple of hours is ideal.

One of the more successful exercises I have used in the past is to ask the candidate to build a simple client for a public API. This tests many things such as working with real world systems, understanding credential management and dealing with network issues and error handling.

Whatever you pick, you want the candidate to be able to create a self contained package or repository, with some basic installation and setup documentation so that you can evaluate both whether it works, and the implementation itself.

Before starting this, as an engineering team you need to create a list of objective criteria that you can score the exercise against. These can include things like checking the documentation is accurate, test coverage, code linting, etc. You can determine your own criteria but they should be as objective as possible so that each evaluator can compare their conclusions.

Once the candidate sends you their completed exercise, the code should be given to several of your engineers to evaluate. This should be done blind so the evaluators only see the code, and they do not discuss the details with each other. This gives you several independent evaluations and avoids any bias. Be sure to instruct the candidate not to include any identifying information in the package e.g. a Github URL or their name in an auto-generated copyright code comment.

Step 4: In-person pair programming

At this point you have done most of the evaluation and believe the candidate has the skills you’re looking for. The final stage is to evaluate actually working alongside you in a more realistic situation. For this, I prefer to meet candidates in person and have them work alongside their potential colleagues.

I have done this stage remotely in the past but have found that it is more effective to meet someone in person. You can then evaluate what they are like as a person. However, this is also the stage where there is most risk of bias. You can mitigate this by involving multiple people from your team so that one person doesn’t have a veto.

In the interests of speed and efficiency, I try and schedule all final interviews within the same week. This may not always be possible but I try to batch them as closely as possible. This makes the best use of your team’s time and means that candidates can get a response quickly.

You should cover all travel costs for the candidates, booking tickets for them rather than making them pay with reimbursement – they shouldn’t have to loan your company their own money! If they have to travel a long distance, offer overnight accommodation, transfers and food. Also ensure they have a direct contact who is available 24/7 in an emergency. You want candidates focused on the interview, not worrying about logistics.

Again, you need to determine what the best approach to evaluating their capabilities is. I have found that getting them to actually work on your codebase is a good way to see how they deal with an unfamiliar environment and start to learn a new system. You can ask them to fix a known bug, or introduce a simple bug into the code and work with them to fix it. You are not testing them on their knowledge, but on how they approach the problem. Whether or not they fix the problem isn’t important.

Remember that this continues to be a sales process. Take the time to introduce them to key members of team, show them around the office and, if they’re not local, the area where they’ll be working. Be sure to show off and explain why you want them to join. This is the job of everyone on the team – multiple people telling them about the company is a lot better than just the hiring manager or CEO!

Step 5: The response

Anyone who gets past step 1 should receive a response to their application whether they are successful or not. One of the worst things about applying for a job is not knowing what the decision was.

The challenge with giving a negative result is that candidates will often ask for feedback and may argue with it. It is up to you whether you want to do this at all, but I usually offer detailed feedback only if a candidate reaches step 3 or 4. Failing step 2 is only for poor spelling/grammar, which you can build into an auto-generated response.

If you are going to make an offer, do it as quickly as possible. Include the key information about the compensation package, start date and anything else you need from the candidate. Be sure to review the legal requirements for a formal job offer first.

Don’t use exploding offers and don’t pressure the candidate. During the step 4 interview, you may want to ask them what their evaluation criteria are and whether they are looking elsewhere. Asking them when they think they will be able to reply to you is probably fine, but  don’t ask about salary expectations.

What not to do

You may notice that certain things are not present in the above process.

• No questions about their background and experience. It is not necessary – you are evaluating them based on their skills and how they apply to them today, not what they claim to have done in the past. That said, in step 4, you may want to ask a few questions about how they may have tackled similar problems in the past, or what interesting challenges they have solved if you are hiring for a very specific problem area. But really, you want to put as much time as possible into designing your coding exercises so they are representative of the problems the candidate would have to solve if they were working at your company. Let them demonstrate their ability, not talk about it.
• No knowledge questions or puzzles. The ability to recall function definitions or solve theoretical problems is not particularly useful for evaluating whether someone can write good software.
• No whiteboarding. You may want to use a whiteboard to explain specific system architecture but there is no place for actually coding on a whiteboard, on paper, or anywhere that isn’t a modern IDE or code editor of the candidate’s choice. Nobody codes in isolation without access to the internet to look things up. Everyone has their own preferred coding environment and the coding interview will likely place them in an unfamiliar setup without their usual shortcuts and window layout, so be sure to make allowances for this too.
• No phone interviews. Again, get the candidate to demonstrate their ability through real tasks, not be explaining what they might do or have done.