My Lessons from Interviewing 400+ Engineers Over Three Startups
Engineering

My Lessons from Interviewing 400+ Engineers Over Three Startups

Engineering leader Marco Rogers (Lever, Yammer, Clover Health) debunks some of the most common recruiting tropes and walks through his four top interviewing practices for startups.

Marco Rogers loves interviewing engineers. Here’s a quick back-of-the-napkin showing just how much: he’s been an engineering leader and hiring manager for the last seven years. Over that time, he’s hired over 80 engineers. To fill each of those roles, he’s personally interviewed approximately 5 candidates on average after initial phone screens. That’s at least 400 interviews, or an in-person interview every work week for seven years — and that’s not counting time dedicated to prep, screening, references or debriefs. Most impressive is that he’s designed a system that has gotten his technical teams involved — and energized — to interview their future colleagues. To him, interviewing is a team priority: engineers on his team who are seasoned interviewers can conduct 12-16 interviews per month.

For those who’ve seen him in action, it’s inevitable that Rogers ended up at a recruiting software startup like Lever. Every engineering leader spends time on building and rebuilding teams, but while at Clover Health and Yammer, Rogers obsessed on designing and refining an immaculate interview process. It was that craftsmanship that helped him hire over a dozen developers at Yammer, grow the Clover engineering team from one to 50, and start to scale up Lever’s technical team.

In this exclusive interview, Rogers starts by debunking some of the common recruiting tropes, explaining why they are outdated or misleading. Then he spotlights his top four interviewing practices — and how they fit within his broader recruiting methodology. Lastly, he recommends the first, low-hanging change that startups can make to retrofit their interview process.

THREE HARMFUL RECRUITING REFRAINS AT STARTUPS

THE REFRAIN: “We test the technical knowledge and skills needed for the role in the interview.”

THE REALITY: You don’t actually know what you need. You’re figuring out what you need.

Across the startups he’s joined (and those he hasn’t), Rogers seeks to change the narrative around mapping relevant knowledge and skills to role. “Even if you’re a serial founder or seasoned technical leader, I’d still argue that you don't actually know what you need in the early days of a startup. There are so many variables, and the equation changes as the market, competitive landscape, customers and other elements shift,” says Rogers. “In the tech startup context, what you’re actually doing is figuring out what you need. You don't start with the answers; you work toward them. Instead, people orient around static sets of skills. So you can recite what it means to build a B+tree. That might be helpful, but not the entire picture.”

I want to make a stronger statement: the skills that people are hiring for are not the skills that they often most need.

So what’s at the root of this misconception? “When I’ve dug into conversations with people about it, I’ve realized that they haven't actually decided that interviewing and hiring is a critical business activity,” says Rogers. “They're kind of just trying to get through it. And because they want to get past it, they look for and lean on patterns, opting to repeat what others have said or done. Of course, they care who they work with, but it’s more about getting people in the door. Then they'll figure it out. That's the attitude. Hiring is not the real activity; it's getting people. It’s a subtle difference that short circuits process for outcome.”

THE REFRAIN: “With engineers, only technical skills truly matter. Soft skills are a nice-to-have.”

THE REALITY: Every engineer must be equally skillful with code and colleagues.

First, Rogers takes issue with the framing of soft skills. “The problem with that framing is that it suggests that you can’t evaluate or teach those skills — that they are inherent qualities that some people have and some people don’t,” says Rogers. “They focus on the ‘soft’ part and forget they are ‘skills’ that can be measured and cultivated in people. I put those people skills at the same level of importance as technical skills. Can you write JavaScript and communicate effectively with business stakeholders? That’s what I need in a growing organization.”

It’s the business’ end goal really makes this an obvious default for Rogers. “First and foremost, we have to address the business reality — what we're really trying to do. If we’re talking about companies that want to grow 10X every couple of years, you have to assemble a really solid team that's going to take you there,” says Rogers. “It’s hard to keep that in mind when there’s the first or next thing to ship. So you have to explicitly test writing code and working together in the interview process.”

Rogers helps his engineers strike balance between the “soft” and “hard” skills by asking them to get answers to the following questions around three key areas:

  • Ego. Does the candidate have low ego? Will they be too protective of their code? Do they have to be right? Does that change how they listen to input or consider other ideas?
  • Adaptability. Do they know that the goal of a business is to grow and change? Have they been a part of a team that has grown where process has had to change or team has had to be restructured? If so, how have they reacted to those changes? Do they seek opportunities for their growth in that change?
  • Technical communication. Do they know the technical skill needed for the role? Have they demonstrated the ability to apply that skill? Finally, can they communicate technical complexities well? Do they recognize that knowledge, application and communication of technical know-how are separate skills to master?
  • Cross-functional collaboration. Can they share multiple examples of close collaboration among individuals across functions, such as Engineering, Product, Design, Customer support and more. Do they have an example of when that collaboration went well? When it didn’t, how did they respond and resolve or bring closure to the situation? Have they proactively identified silos in an organization — and what have they done about them?

THE REFRAIN: “We’ve picked a few people to interview with them — let’s get it scheduled.”

THE REALITY: Thoroughness and consistency is paramount for optimal, unbiased results.

When Rogers talks about an interview program, it’s a crafted process. “So if you come on site with us, you're going to have four sessions. That’s our ‘right’ number, because we don’t want to ask too much time from candidates and be grueling, but we also need to have enough data points. The four sessions include three evaluations sessions with the team, each an hour long. Then there’s a session with the hiring manager, usually me, so I can answer any questions about how the team works,” says Rogers. “It’s designed this way so candidates can talk to a number of people: engineers of every tenure, so they can paint a picture of the day-to-day and their career path. We get to put you through your paces, and so candidates have several opportunities to display different skills. Each session has a fixed exercise or questions to ask, and then some time for unstructured dialogue. There’s a balance to strike here. The consistent structure and exercises help to reduces bias. While the open conversation makes it human. We’ve gotten feedback that this agenda and duration gives candidates sufficient time to start evaluating us as a potential fit.”

Lever Director of Engineering Marco Rogers

STEPS TO TAKE TO MAKE INTERVIEWING A CRITICAL BUSINESS STRATEGY

Rogers has designed a system to make interviewing a hallmark experience for candidates and a strategic advantage for your startup. Here, he shines a light on four components, but recommends using them together, so their impact compounds. Let’s get started.

Interview in three-person groups.

A go-to technique at Yammer and Clover, three-person interviews are one of Marco’s most long-standing and valued interview preferences. He saw the benefits of them early in his career, but more so, observed how traditional interviews are broken. “One-on-one interviews tend to lead to a unidirectional candidate-and-interviewer dynamic. A person asks a question, the other person tries to answer it. Rinse and repeat,” says Rogers. “We know that that’s not a great candidate experience, as they come away feeling grilled or discouraged after trying to answer a bevy of questions. Interviewers can also get disenchanted, when, instead of following the thread of the conversation, must cycle through questions like a more rigid, rote activity. From the angle of both parties, the interview can lose its more organic, collaborative aspect.”

The quickest and most effective way to move beyond that poor interview experience is to add another node: a third person. “Add another colleague into the mix as a second interviewer. I’ve found that it not only brings a higher fidelity and signal to the conversation, but also breaks down the flow and nature of the conversation,” says Rogers. “Sometimes a person will be talking to the candidate, and other times, they’ll address their colleague. I find questions and answers bouncing between three points opens up the discussion and evaluation much more.”

Three-person interviews generate more signal and meaningful meandering than traditional 1:1 interviews.

To show how powerful the three-person interview can be, Rogers’ shares a few scenarios, which he’s experienced and anonymized, that drive home the benefits of this interview approach:

  • It subtly helps reduce bias. “Say you send in two interviewers: one's a man and one's a woman. You'd be surprised how often the candidate will only talk to the man, even if the woman asks the question. And how there will be no eye contact or engagement with the woman,” says Rogers. “But, if it was just a woman interviewing the candidate, the candidate would naturally look at her as the other person in the room. This bias wouldn’t surface, nor might she sense it at all or to the same degree. And, to be very honest, if you send in a single interviewer who is a man, it’s unlikely he is going to pick up on that situation. With a three-person group, there’s a way to compare notes and realities.”
  • It splits the acts of engaging from observing. “Take technical exercises. They’re timed and, in the context of an interview, can be very stressful for candidates. Say a candidate is bombing, and the interviewer is not helping them out. It may be because the interviewer is tough or is busy being a procter, observing the exercise underway,” says Rogers. “When you send two people in, one of them is generally observing more, as the other engages. As people talk, those roles of engager and observer swap. In this case, the observer might see that the person is bombing earlier and step in to right the ship. I’ve heard an observer jump in quickly and say, ‘Hey, it sounds like you might be getting bogged down’ or ‘Hey, let me throw something in that I think might help.’ It’s not only a relief to the candidate, but can salvage an interview and bring it back on track. It’s also easier to track time with two people. But when you’re one interviewer, who’s doing both the engaging, observing and timing, it’s a challenging to be present and do the job right.”
  • It simultaneously trains your more green interviewers. “When I talk about two interviewers in every session, there's a couple of parameters. At least one of the interviewers has to be practiced on the exercise or type of interview we’re conducting and often the other is learning it,” says Rogers. “ That way you're naturally training the interviewer at the same time you’re vetting a candidate. Interview training and modules sometimes work, but I’ve learned that there’s no substitute for the real thing. Plus, when you mix people who are new to interviewing with many seasoned interviewers, they’ll pick up a lot of different techniques and styles over the course of a year.”
  • It elevates the perspective of less experienced employees. “For many, it’s most intuitive to send your most senior people to evaluate someone. The belief is that a more experienced person will be able to push candidates as far as they can go and assess their seniority,” says Rogers. “Personally, I haven’t found that correlation. And more problematically, organizations get a blindspot with that approach. Newer engineers come out of interviews with a keen sense of if someone is better than them, what they might be able to learn from them, and whether the person has manager or mentor potential. As a leader, you get a better sense of career tracking with this kind of intelligence.”
It’s tempting to only tap your senior people to interview candidates. Less experienced people get a different signal — namely, if they think they can learn from and be led by a candidate.

Everybody — the entire team — interviews.

This interviewing principle is one that Rogers has seen get pushback. “For some, it’s an issue of scope and something leaders and recruiters should be doing. For others, it comes down to a concern that not everyone can represent the company in an interview. People also say that it’s too costly to have everyone trained and involved,” says Rogers. “The one most people have the hardest time with is cost. But it’s going to feel costly — and it should. Interviewing is a critical business activity. It's like saying, ‘Hey, we're having engineers write a lot of code. That feels costly.’ That's kind of the thing we need them to do, and that's how I feel about interviewing. I need the team to help me understand if we are bringing on the right people. These are people you’ll be working with every day for years. I think that’s worth a few hours of your time periodically.”

Startups are either growing quickly or want to scale. Either way, having more people being able to interview candidates skillfully and efficiently is an advantage. As the engineering leader, Rogers sees it as his responsibility to develop this skill in his team. “I put myself in a position to evaluate how the engineers on my team are doing in interviews. In the early days, I pair myself with new interviewers in the three-person sessions. In those situations, I’m not only evaluating the candidate, but observe the engineer’s ability to interview,” says Rogers. “When we do a roundup with all the interviewers, I ask each person to reflect back to the group how the sessions went. I can ask more questions, probe and help them unpack what happened. This paves the way to offer feedback and suggest other ways to get through interview challenges.”

The other advantage in having everyone interview is that it’s easier to identify members of the team who are comfortable, confident interviewers — people who the engineering leader can tap to serve as mentors to make less experienced colleagues better interviewers. “I get to know which engineer is skilled with candidates, both from feedback from the roundup, as well as being in the interview with them. For example, I keep notes on who’s good at conducting the technical exercise or a certain series of interview questions that we ask,” says Rogers. “With that context, I have an understanding of interviewers’ strengths and who can help their colleagues improve their interviewing skills as they get more sessions under their belt.”

Your entire team should conduct interviews. Everybody. If you don’t want some people to interview, ask yourself why. If you’re worried about how they’re representing the company, there’s a bigger issue at hand.

Commit to doing more interviews.

Rogers is a firm believer that initial interview screens and filters are broken. “Conventional wisdom is that you can look at a piece of paper and tell if you are looking at the top 10%. At every turn, I’ve found that to be incorrect. Talk to engineering leaders about how their team members are doing and how they thought they’d perform when they hired them. Increasingly performance doesn’t match up with pedigree. Maybe they attended an Ivy League school or worked at Google, but they're not coming through,” says Rogers. “Conversely, you have that person that's really kicking ass, but when you look at their profile, you didn't see that. You didn't anticipate how high performing they are today, because it’s about if they had the right opportunity, motivation, and investment in their work. This not only has a lot to do with the individual, but the structure that a company provides to bring the best work out of their people.

All of those elements don’t really show up on a LinkedIn profile or resume. Those things are about how good someone is at getting a job, not how a person will add value to your org. Profiles are only one data point, and you need more.”

Given that screens and filters are counterproductive, Rogers sees the only way to fix them is to do many more interviews. “From talking with my peers, most bring about two candidates on site for an interview in order to hire a mid-level engineer. I believe that there should be many more interviews—at least ten for every hire made,” says Rogers. “If you’re at an early-stage startup and you’re going to be hiring a lot, set up a structure that can handle interviewing many candidates. And by hiring a lot, I’d say at a clip of 10 or more people every six months. If you’re only hiring about five people in a year, you can get away with doing fewer interviews — but I wouldn’t recommend it. One year, I hired 50 engineers. I couldn’t have done that in a high quality way without a system that let me interview about 500 candidates. Create that interview-intensive system, stick to it, learn from it and you'll make better hires. That's my stake in the ground.”

Rogers readily admits that partnering with an in-house recruiter has been the only way he’s been able to create a hiring system that can support many interviews. “Even early on — especially early on — you need someone collaborating with you on the scale and scheduling of all the interviews for a hiring process. At my last three companies, I worked with a recruiter, a colleague whose job it was to help set up the interview program I designed and make it consistent,” says Rogers. “It’s on me to make sure that the program works the way that we're specifying today. That’s another reason why, as the hiring manager, I not only do a final evaluation of the candidate, but also get feedback on how the interview process went, and where it could improve. This is the information that helps us make the interview process better and more efficient which, in turn, helps us interview more people effectively.”

You want a top 10% developer? The most effective way I’ve found to get to the top 10% is to interview and say no to the other 90%.

Round up and huddle.

Rogers is a champion of roundups and huddles to wrap up an interview process. It’s the final piece of the puzzle, and the dynamic exchange of information that brings critical context that is rarely captured in written reviews. Huddles take coordination though. “When I talk to other people about huddles, they're not sure it’s worth it. It’s mostly because of logistics. To bring together everyone who interviewed candidates for a role is a lot: six engineers from three two-interviewer sessions,” says Rogers. “They are 30 minutes long, but once you get good at them, you can get them down to 15 minutes. I highly recommend that every team takes the full half hour if they are just incorporating a huddle into their hiring process.”

Here are the objectives for the huddles that Rogers runs:

  • Get an initial yea or nay. “So Lever has a rating, that says hire or no hire — it’s broken into a four-point scale: strong hire, hire, no hire, strong no hire. It’s intentional that there is no ‘maybe’ rating — people must make a choice with their evaluation. That rating is good, because that's the place to start. You said no hire, let's talk about why. Or you said we should strongly hire this person, let's talk about why,” says Rogers. “I like starting with this for two reasons. First, engineers love rubrics and rooting their rationale in a system. In this case, we root to examples, not just preferences. So, in asking about engineering tradeoffs, I want to know if they understood the tradeoffs in NoSQL databases versus a candidate’s preference for MongoDB. An ability to make a confident statement is not as important as an ability to give me a critical analysis. Second, making a ‘hire’ or ‘no hire’ decision upfront means starting with the end in mind. Beginning with an interviewer’s top-level assessment gives context to the rest of their evaluation.”
  • Amplify signal. “As a hiring manager, much of what I’m doing is making a determination about how strong the group’s signal is out of a particular session. I ask each interviewer to go through their session’s exercise or line of questions, and ask how it went,” says Rogers. “If it seems as if the candidate was nervous and really bombed, I don't know how much I'm gonna get out of reading the description of that session in the written evaluation later. So I'm not going to evaluate that data point as highly. But if an interviewer says they had a great session, and that it was really engaging, I’ll pull on that thread to learn more. And so, the huddle is really about me getting a sense for, how did we do in this evaluation, how did the candidate do in this evaluation. I want voiceovers to keep in mind before I re-read and interpret the written feedback.”
  • Surface skill bias. “No matter how repeatable and standardized your interview process gets, it’s driven forward by humans — and with that comes judgment and bias. I’ve seen how that’s difficult for people, because they want to believe that they can control their biases, especially if they’ve identified them. But just because you know a bias, doesn’t mean you can remove it,” says Rogers. “So when it has to do with candidates and interviews, we use the huddle to call out our biases. For example, some engineers discount candidates if they don’t name their variables well in a technical exercise. The truth is that’s really common. So it’s important to level: ‘Is naming variables well in code really an important thing? Yes, it is. Is it a really coachable and fixable thing? Yes that’s true. So it shouldn’t disqualify people. That’s why we do code reviews. We teach this.’”
  • Spotlight who’s making the decision — and how. “This is an important time for hiring managers to set — or reset — expectations on the final decision. Acknowledge the value of each interviewer’s contribution, but also convey that owning part of the process isn’t the same as owning the decision. Show gratitude for every input, but remind the team that you’re making the call about whether an offer is given,” says Rogers. “The flip side is that you should bring visibility to your thought process. In a huddle, I try to give the team more insight into how I'm making decisions. I’ll tell them; ‘Hey, this observation that you made is a really strong signal; I'm weighing that highly.’ Or ‘Hey, I'm not sure how much we should weigh that comment. They might have been really nervous.’ Regardless, I like to tell my fellow interviewers that we should talk if what I'm saying doesn't sit well with them. It’s important that the team doesn’t feel as if I’m making poor decisions.”
The huddle helps bypass the biases that traditional screens and filters reinforce. It’s also a prime opportunity for leaders to shine a light on how they make decisions.

TYING IT ALL TOGETHER

From initial screens to session construction to debriefs, the way startups interview candidates needs to improve. To start, don’t fall prey to interviewing tropes. Recognize that you may not know the qualifications you need for a role, that engineering candidates must meet high expectations on both technical and “soft” skills, and that you need a thorough, standard interviewing system. That regimen should start with three-person interviews, which is not only a more effective way to evaluate candidates, but also a valuable opportunity to develop your pool of interviewers. Everyone should have an opportunity to interview — no exceptions. If you are anxious about that decision, dig into why. Commit to doing more interviews; it’s the best way to bypass faulty filters and screening. To get the top 10% you need to interview the other 90%. Finish each onsite interview with a huddle. They generate dynamic reads on candidates, and bring visibility to a leader’s decision-making.

“So what’s the first step to level up how your startup interviews candidates? There’s actually a step zero, which is asking: are you getting enough different data points in an interview process? If you need a more powerful signal, try this methodology. If you’re a startup that has its founders interview a candidate and then make the offer, involve more people in the interview process and debrief in a huddle. The highest leverage will be getting those interviewers rounding up to compare notes,” says Rogers. “When I talk to people about interviewing, I notice that a lot of companies are just still doing the interview style that Microsoft pioneered in the 90s. That’s what much of our industry still does. But that gets us 90s Microsoft people. If you’re a scrappy startup, your hiring expectations and approach must be different. This system has not only strengthened our candidate pool, but also the cohesion, versatility and diversity of my engineering team.”

Photography by Bonnie Rae Mills.