Weeks after eero launched its eponymous product, Head of Beta & Special Programs Kelly Neary and Head of Product Paul Nangeroni finally caught their breath. Thousands of customers ditched rickety routers in favor of eero’s home WiFi system. Fortune, The Wall Street Journal, USA Today, The Verge and others weighed in with positive reviews. If you ask Neary and Nangeroni to pinpoint the seed of eero’s successful launch, they’d respond: our beta program.
For the six months leading up to the eero launch, it was all-hands on deck at the startup, but Neary and Nangeroni led the effort. Neary hails most recently from Nest, where she drove its early beta program efforts and later launched a network of professional installers across seven countries. At Apple, Nangeroni led hardware program management for a range of products including multiple generations of iPod Touch, iPhone 5C and the Apple Watch.
In this exclusive interview, Neary and Nangeroni share why betas are fundamental for a successful launch. As contained dry runs, these tests not only surface insights that help refine a product release before its launch, but they also lay the groundwork to engage early adopters and groom brand champions. The duo segments the beta process into three parts — the lead-up, launch and loop-back — and imparts tips to enhance each stage. Any startup that is gearing up to go-to-market would benefit from the insights from the eero launch.
The Lead-Up to Beta
From the start, eero knew that a beta program would be particularly important for its launch. There was not only hardware and software to test in tandem, but also the guarantee that reliable WiFi could be promised to all people, regardless of the configuration of their home. The beta program would put eero’s technology and brand to the test. “Because WiFi's so integral to our lives today, we really had to learn how best to support the product. In this case, you can’t run a beta, play with it for a few hours and put it away when you're done for the day,” says Neary. “We went in knowing that it’d always be running in the background of our testers' homes, so we had to be ready to support them quickly and comprehensively, seven days a week.”
From a beta standpoint, that meant there was a higher bar to hit to get the most from a test. “We're asking people to take something that is a physical piece of infrastructure into their home. To get useful data for us, it had to work well enough to be actually useful to them,” says Nangeroni. “So we decided to think as carefully about our beta as many startups do about their actual product launch. It was one of the reasons why we brought on Kelly. Drawing from her experience at Nest, she jump-started the beta and knew how to refine and scale it.”
Rally internal, cross-functional buy-in first.
Be intentional about getting each function at your company on board from the start. “Don’t assume that because your team uses the product and is behind the beta that they are logistically ready to support it. It takes a lot of momentum to organize each team internally. It goes beyond getting dev work done or basic support tools in place,” says Nangeroni. “A beta should be dry-running a launch, as if the product is about to go live. One way that we made it real for our employees is that we asked them to call in or email the support team if they had an issue with their eero, not just walk over to the support desk. They sampled the experience that our beta testers would. Betas takes this type of nuanced planning and rollout.”
Especially for early-stage companies, it’s key that pre-beta meetings convene cross-functional participants. For eero, it was an early examples of everyone working closely together. “You want stakeholders from each team who might be either presenting or consuming feedback from the beta group. You have to look at it as if it's not really a meeting about the beta program. This is the practice for production,” says Nangeroni. “The cadence will change as you get further from the full, public launch. Weeks after production, we continued to convene every day. Six months from then, it’ll be every week. The meeting cadence may change but the agenda doesn’t. The beta's a great way to feel out exactly how that process should be structured.”
Your beta is where your ‘maybe’ gets clarity. It’s the trial run for testing your support operations in-house.
Design your beta with diversity top of mind.
For eero’s beta, the team reviewed detailed profiles of applicants who were referred by friends and family. The prospective betas filled out an application that allowed eero to select a group that had very diverse representation of what it believed its true customer set would be. This helped Nangeroni and Neary get exposure to the broadest user base possible to identify any big road bumps that might emerge before the official launch.
“We’re building a product to go into people's houses, but homes are as unique and different as the people are themselves. Relying solely on employees to test doesn't help when everyone in the company lives in a tiny little 1-bedroom apartment in San Francisco,” says Nangeroni. “It’s very hard to approximate how WiFi performance should look in a 2,000 square-foot suburban Midwestern household.”
According to Neary, here’s how eero designed its process and questions with diversity in mind:
- Devices. “eero had to work well with all connected devices. Phones, laptops, tablets, wearables — you name it. We wanted to cover our bases by choosing homes that would test each of the devices on our list.”
- Homes. “We very deliberately chose homes in different states with different ISPs. That in itself brought a lot of variability. We also seeded our sample group with homes of different building materials, which impact WiFi reach and strength.”
- User expertise. “We anticipated that our customers would exhibit a range of tech-savviness. Not all would be super home-networkers.”
In addition to building in variety into your sample, structure questions to unveil diverse experiences in a subtle way. “Instead of asking how tech-savvy users are, we asked about their profession, the kind of modem they use and the type of devices they have running on their networks,” says Neary. “If they didn’t know the kind of modem that they had, that was fine and its own, valuable data point. Taken together, they became strong — and honest — indicators.”
No matter how expansive your diversity parameters, expect to have gaps that your beta group will help you fill in. “We had someone who wanted to get WiFi in his treehouse. So in the beta, we tested it out. We didn’t intend to be faced with that type of home, but I’m glad we did. It’s one less circumstance that could have surprised us in the official launch,” says Neary. Nangeroni adds, “Again, that's the value of tailoring questions to your value proposition. So when we asked if beta users had dead spots in their home, cases like treehouses would come out of the woodwork. Those types of instances constantly — and helpfully — tested our understanding of our target demographic.”
Make your beta polished, not perfect.
The big question is when to take your test beyond company walls. According to Nangeroni, there’s not a strict timeline, but a balance that must be hit before you beta test.
Don’t wait until all is buttoned up to beta. You’ll lose the chance to really iterate. But don’t go in fighting a multifront war either.
Your goal should be to split the difference, where there are no outstanding deal breakers for day-to-day use, but enough rough edges that’ll prompt meaningful feedback. The question to start with is: what is the indispensability of the product that you're testing? “You’ll have a different answer if you are testing a fundamental piece of infrastructure or something that’s hooked up and used for a couple of hours a day,” says Nangeroni. “Here’s how I gut-check: if you have immature hardware with plenty of known bugs, software that has major stability and performance issues, or a green support team without the right tools, take pause. Any one of these on its own, and you’re fine to start. Even two is is ok if you have a tolerant set of beta users. But any more than that and it will be difficult to gather meaningful insights above the noise created by the product immaturity.”
The Beta Launch
From end-to-end, eero’s beta test lasted six months, culminating with the product’s public launch. If knowing when to start a beta is set based on product maturity, knowing when to end it is much more well defined. “A conservative rule of thumb is to set product launch for at least three weeks after you lock production candidate software”, says Nangeroni. “This gives sufficient runway to evaluate the qualitative and quantitative metrics on the shipping code and to have confidence that the product is ready to release.”
That said, Nangeroni recommends that beta runways should be measured less in months and more by metrics. “Give yourself a reasonable amount of time to collect feedback and funnel it back into the product prior to launch. Ideally, the beta should end when you’ve made the necessary adjustments to hit the level of performance, stability or speed that your metrics demand. Your beta customers — who are living the day-in-the-life with your hardware and software — need time to help you get that last mile.”
So does the beta stop once the product is launched? Nangeroni views the process as more fluid. “I personally believe that betas should run forever. It may change form. You may swap out beta customers here and there, but the feedback loop should persist,” he says. “The value that you gain from being able to collect these user insights is something that product teams should want for infinity.”
A beta test is more ping pong than rocket launch. It’s a continuous exchange, not a release that’s only observed.
Roll up your sleeves — you are your best tool.
Running a great beta program takes a lot of work, especially for early stage companies who lack established processes and tools afforded to larger corporations. “Don’t underestimate how much manpower a beta program is going to require,” says Nangeroni. “Say you want to deploy some new hardware in the field, but it showed up with the wrong software on it. Maybe the tool that's supposed to automatically update is broken right now or a server is down. There will be countless unforeseen issues that poke holes in even the best laid plans and force a lot of brute force dedication from the team to make sure everything keeps moving forward.”
For eero’s Beta launch, the team touched each part of the process. “Three people were dedicated to leading the beta program, but it was an all-hands-on-deck, cross-functional effort,” says Neary. “I was packing boxes and filling out address labels. We took notes and funneled them into call logs. As a company grows, there are more tools, but you’re your best tool, especially at the earliest stages.”
Iterate and improve your beta real-time.
Companies that get the most out of their betas improve the user experience and the test itself as they’re both in process. That starts by integrating them during the product development process as if they're an extension of the team. “The beta is an opportunity in several ways. First, we want anyone who comes in contact with eero to have a delightful experience. Our betas were our first evangelists, so we wanted to make sure they were super happy,” says Neary. “If served correctly, this not only spurred product development but foundational brand building. We knew we did something right when our betas were chomping at the bit to post on social as soon as we launched. They’re our biggest advocates.”
According to Neary, here’s how eero turned its beta users into its leading champions:
- Declare frequency of outreach upfront. “Before our beta group signed on, we made it clear how often they could expect to hear from us. We let them know that we'd be in touch once to twice a week to hear about everything, from a feature request to a bug they’re finding.”
- Seek process not just product improvement. “Our beta testers not only receive questions about the product, but also about the beta test itself. We ask them how the beta is going, what days and hours they’d like to be contacted, if they prefer email or phone for correspondence and more. We track NPS for the beta program separate from the product NPS.”
- Don’t close the loop; close their loop. “When we had new deploys to share, we'd reach out to the people who were expecting that fix or who had experienced that problem, letting them know that we were solving the bug. Open lines of communication are super helpful only if they're relevant to the users who initiate them.”
- Cycle in fresh perspectives. “Don’t get complacent towards the end of your beta. Use the final two weeks to integrate a new set of beta users who can truly bring fresh eyes. When we did one of our final big software upgrades, we pushed it to a mix of some of the earliest beta testers and those who had never seen the product or experienced the beta. We continuously introduced new people into the setup process to get new perspectives to complement feedback from experienced beta testers.”
- Pepper in postmortems. “We did a postmortem after each release to note what worked and should be done differently. What was particularly effective was not waiting until the end of the beta to do postmortems, but doing them incrementally throughout the beta test.”
We thought speaking to betas twice a week was high-touch. Their feedback was that they’d be willing to spend more time with us.
Calibrate feedback from friends and family.
Many founders debut their product with peers and relatives, who they often count among their earliest champions. While this group of people shouldn’t necessarily be avoided, they should be clearly understood in the context of a beta. “A beta is about getting really useful and valuable user feedback in a contained, organized fashion. Friends and family have their pros and cons,” says Nangeroni. “They are great for security, confidentiality and their inherent tolerance for the immaturity of the hardware. However, they’re really, really bad at giving honest answers about how they think the product is doing. It turns out my mom likes just about everything I do.”
Nangeroni and Neary knew they weren’t getting unbiased feedback when eero’s friends and family logged an astronomically high NPS score. “This is when you need to put survey feedback to the test. You can’t rely on it alone to decide whether or not to address a customer need. Instead, dig into each person’s free-form responses and sanity-check whether your product is performing as the customer believes it to be. In eero’s case, we also compared survey responses to the data we have on network health to unearth more context,” says Nangeroni.
One way to guard against overly bullish reviews from friends and family is to rewrite survey questions. “Veer away from assessments that ask to give a ranking on a scale from 1 to 10. Instead ask questions that have a forcing function. For example: ‘If you had to alter one aspect about the product, what would you change?’” said Nangeroni. “Tailoring questions to avoid drawing out bias maximizes the utility of the survey. Even if you’re starting with friends and family for strategic reasons, you still want to get everything you can from their answers.”
If eero was double its headcount at the time of its beta launch, it might have brought the beta in-house. “We eventually opened up a generalized link for people to apply to be involved in the beta program, but that was only distributed through employees. We screened external beta users closely to keep the product under wraps, especially considering press embargoes,” says Nangeroni.
Buoy your expectations on confidentiality with trust.
Neary and Nangeroni recognized that even a partially external beta was a risk, but it was worth taking to get the best first generation eero out the door. “We did take a few steps to mitigate any misstep, but in the end you have to trust your beta group. First we reviewed each beta tester’s personal and home profile, regardless of who referred them. Then we clearly communicated the process and end goal, which we emphasized required their discretion,” says Neary. “And then we reminded them again. In every box of eeros that we shipped out, there was a letter from [eero CEO] Nick thanking them for their participation and reminding them to not share any details of the experience. ”
Looping Back After The Beta
The final critical step before moving from beta to public launch is circling back to analyze feedback with your team and close the chapter with the original group of early users. Your beta has wrapped up and you’re staring at a wall of feedback. “We were fortunate that we had lots of responses from our testers, who loved talking to us. The goal is to assign impact. You do that be seeing how many of your target users will be impacted as a percentage of your total customer base,” says Neary. “A few people with a severe issue may be a higher priority than if everyone has a low-impact problem. I went line-by-line through comment sheets categorizing and ranking. Then it was time to share the feedback with the larger team.”
Guide engineers from the frontlines to the bottom lines.
After the beta finishes, there’s a lot of iteration on feedback relayed between support teams and engineering that must happen before the official launch. “It is very difficult for engineers to take action from a generalized response. Often, they want to hear from the beta user so they can reproduce it locally and get clear on the exact customer need,” says Neary. “Our solution involves triage meetings, throughout which we aim to tighten up the flow of communication between customer support and engineering on user feedback.”
Streamline Slack channels to help consolidate communication. “Initially at eero, the team had different Slack channels for customer support issues, employee ‘dog food’ issues, email and SMS,” says Nangeroni. “All these forms of communication made it difficult to focus on what's important. While the team had excellent response rates to reported issues — often within minutes — it was far too reactive, resulting in engineers being distracted from more critical work and bypassing production support flows.”
Eventually, Neary and Nangeroni started to funnel all communication from chat platforms into a standing meeting where support could review with engineering at a regular cadence. “It’s hard because engineers don’t want to be shielded from timely, specific issues. Support must help triage in order to roll up the issues, focus on the big ones and deliver to engineering.”
It seems counterintuitive, but steady triage is better than reactive responses in the long run. “I can't underestimate how hard this is, because when you put in place this type of process, it feels like everything's going slower and that you're being less nimble and responsive,” says Nangeroni. “But the alternative is having engineers not hunkering down to solidify the features that they're supposed to be shipping. If they're responding to ad hoc requests, the support team isn't validating their tools to be scalable because they also start to operate reactively. It’s a vicious cycle.”
One way to shorten the distance between the engineers and the beta group is translating and delivering technical questions on the development team’s behalf. “We would ask the technical teams what they wanted to learn more about, and then it was up to us to structure those questions in a way to get the type of feedback they'd wanted back,” says Neary. “If the questions were deeply technical, they wouldn’t make sense to the average tester. We could help by translating questions from the technical team.”
Every email, unboxing experience and interaction for the beta user is one more opportunity to fine-tune our brand, our messaging, and who we are as a company.
Honor all feedback with gratitude.
If a beta user gives you what appears to be irrelevant feedback, still thank them for it. “It’s not only important to be grateful of their time, but you must remember they’re deep in this process with you. Our betas took the time to take down their entire network and set it up again from scratch. The least we could do is show some deference to their feedback,” says Neary. “We definitely record all feedback because what may seem trivial at the time could become common among a critical mass of users. For example, when we first sent eero systems to beta testers we didn’t have a ‘start tab.’ We heard a few testers say they didn’t know exactly where to begin since all eeros look similar. After recording this feedback a few times, we realized we needed to better guide users so they’d know where to start.”
Offer closure and more exposure.
If you’ve done your beta correctly, your early users are invested not only in your technology, but also with your brand on an emotional level. “Part of success is developing a sense of community with your beta customers. It takes a lot of effort to reach out, but over time you start to develop rapport and relationships with these individuals,” says Nangeroni. “When you’ve established that trust, people are not only satisfied with what your product does, but excited for where it’s going.”
Nangeroni and Neary recommend closing the chapter with the beta and offering options for early users to stay engaged. “Start by sharing the milestone with them. Launch day was a defined achievement to celebrate together as well as another opportunity to check in. It’s very much like a graduation,” says Neary.
When it comes to continued engagement of your beta users, have clear options that allow them to stay engaged. Eero has defined paths for three distinct groups:
- Community members: “These individuals are the most vocal, helpful and typically the most tech-savvy beta users. We can work easily with them on a 1:1 basis to fix issues. Whether they are friends or third-degree connections, we know their home layout and their profiles. We have a strong relationship.”
- Target customers. “These betas represent the profile of the user that we are seeking to attract and retain, per our one-year roadmap. If we know we want to target a certain type of home, user or ISP, we bring that contingent of our beta group into the fold.”
- Corner cases: “Our customer support team gets a lot of calls from very different types of homes and use cases. Each one gives us a better sense of the edges of our customer base. For example, WiFi for tree houses. If those customers are open to working with us on a 1:1 basis, we're going to invite them to continue to work with us.”
One beta had a connected device that didn’t work well with eero. In the end, the issue was with the device, so we bought him a new one to thank him for his time.
When eero launched in early 2016, it had already vetted the product with hundreds of beta users. The support team knew their first names and the floorplans of each home. Many of the betas are vocal champions of the WiFi system. The outcome was the result of careful preparation and diligent execution of eero’s beta program. As leaders of the effort, Nangeroni and Neary enhanced all three parts of a beta test: the lead-up, launch and loop-back. At the start, it’s as critical to rally a diverse set of early users as a cross-functional crew to support them. When launching, iterate and improve not only the product in real-time, but also the process by which you receive feedback. Finally, keep your engineers responsive — but not reactive — to beta requests and create new ways to engage your loyal beta-testers-turned-brand-champions.
“A beta is more than making a product launch-worthy — it’s a good trigger to weatherproof all parts of the organization for scale. Does the tool you’re using to send email to users work? If they have comments, are they getting captured in customer support? Betas are end-to-end tests for your product and startup,” says Neary. Nangeroni adds, “Don’t underestimate the work it takes to execute an effective beta. It’s takes a lot of grit, focus and commitment. This isn’t the sort of responsibility that you can task a guy on your team who's already doing three other jobs. It takes a serious dedication from people on every rung across each department. To be fluent in beta testing is to be conversant with the customers that’ll help build your future success.”