Creating a user research program that drives itself · Segment Blog
In a scrappy B2B startup, user feedback is super valuable, but guerrilla research won’t cut it when you need a more targeted group of users. The Segment Design team found the users we needed and developed an automated process for recruitment and coordinating interviews using our own product and a few integrated applications.
Guerrilla research includes a range of fast and inexpensive techniques for designers and UX researchers (often the same person) to observe how users engage with products in the wild. Rather than recruiting participants, these ad hoc experiments are usually done on friends, peers, or strangers in coffee shops. But what if the users of your product are other businesses? Even worse, what if your product is (gasp) technically complex for a niche audience? The ‘check-out-my-mixtape’ methods that work well for consumer-facing products will get you a lot of blank stares.
Sourcing users for research and feedback is particularly challenging for the Design Team at Segment because our product is deeply technical. While the customer feedback we gather through success tickets (Zendesk) and NPS surveys is valuable, we needed a way to explore our user’s behaviors and needs in more depth.
To do this, we designed an automated workflow to recruit participants for a UX research program and and to coordinate testing and feedback sessions throughout the product development process. This article explains how we developed this workflow and describes the final system, which you can borrow from and improve.
Our First Attempt
Preliminary Goals:
Find users willing to give us feedback.
Interview users to better understand how they were using our product.
Identify pain points in the current experience of using our app.
Do all of the above at regular intervals without spamming our customers!
Preliminary Solution:
Our initial process was inspired by Mesosphere, who wrote about their experience bootstrapping a UX research program. We used Customer.io to email new users, asking if they would be willing to join our User Experience program.
Since we shared a Customer.io account with other teams at Segment, we could specify that only users who had not been recently contacted by other teams would receive the emails. (Sorry, thirsty UX researchers: empathy for your users includes not spamming them.) Recipients of the opt-in email would fill out a Google Form, which recorded their email in a spreadsheet. We would then periodically email our pool of opted-in users with invitations to remote or in-person research.
The upside of this process was that it didn’t take very long to set up, and the Google Suite tools were free. But as the pool of participants grew, it became clear that the time-intensive nature of manually managing the spreadsheet and sending emails wasn’t scalable. Recalling the proverb, ‘Physician, heal thyself,’ we took a Design Thinking approach and treated our user research program like any other user experience challenge.
Version 1.1: Automating the Workflow
We want to spend our time researching and designing, not sourcing users and coordinating sessions - Segment Design Team
For this iteration, we took a much more rigorous design thinking approach. We approached the process holistically, considering both the researcher and the participant as users when iterating on our process.
Jobs To Be Done:
As a product designer, when I am exploring or validating an idea, I want to be able to interact with users so I can learn more about them and incorporate this understanding into the product.
When I am doing user research and testing, I want to be able to find users easily, so I can spend my time learning more about them rather than coordinating the process.
When I am engaging in user research as a participant, I want to be able to give feedback quickly and easily, so I can move on to my primary responsibilities.
Problems:
The first iteration of our user research recruiting and coordinating process required too much manual input on both sides. The opt-in experience for our participants was not ideal, since we were sending an email which, somewhat paradoxically, requested that they enter their email in a Google form. The pain points on our side centered on the way email addresses of participants and the records of when they had been contacted were trapped in spreadsheets and not accessible in other tools.
What worked well from the last iteration:
Allowing users to explicitly opt in to our program made sure that we weren’t spamming people who weren’t interested in participating.
Solution:
We began to explore ways to streamline the recruitment and coordination process. When we mentioned the project to our developer counterparts, they were aghast at the existing manual process. Apparently repetition is the Comic Sans of engineering.
As we ideated on how to automate the workflow, a crazy idea emerged: What if we used our product, a platform for customer data, to collect and manage other kinds of data?
Sidebar: We did use our own product as part of this solution. No, this is not a sales pitch. Yes, there’s a free plan that should let you achieve this workflow.
Roll your own UX Research recruitment system in 20 minutes
1. Fill the top of the funnel
We continued to use Customer.io to send a triggered email to new users who signed up for Segment and ask if they’d like to opt in to the UX Research program.
2. Tag users who opt in
We designed a landing page and launched it with WebFlow. When users reached the page, we used Segment to assign an attribute to their user IDs with a tracking API. In this case, we assigned the attribute ux_research_opt_in=true to the user ID. If users chose to opt out of the program, we simply changed this attribute to ux_research_opt_in=false to remove them from the UX Research program without unsubscribing them from all Segment emails.
Pro tip: If you’re a designer who isn’t also a developer, get some help from a friend with this step. If you’re a designer who doesn’t know any developers, you might have some trouble shipping products.
3. Invite participants to sessions
Using Segment meant that the the attribute assigned to the participant’s user ID was available in hundreds of third party tools. Having the opt-in attribute associated with user IDs allowed us to have consistent cohorts across various channels, in this case Customer.io for emails and Intercom for in-app messaging.
We planned to contact participants no more than once a month and sent them Calendly links so they could self-schedule remote or in-person sessions at their convenience for the subsequent two weeks.
Pro tip: Create an email alias for your UX Research program. Otherwise replies and out-of-office messages will end up in your inbox. This also allows you to create a research and testing calendar that the entire Design Team or broader organization can view.
In addition to the remote and in-person sessions, we also sent out surveys and exercises like card sorting and tree testing asynchronously via Verify and Optimal Workshop. This was important because it allowed users who live in different time zones across the world to share their perspective.
4. Show some love
Finally, we sent thank you gifts to our participants using Printfection, which lets customers select from various Segment swag and handles all aspects of fulfillment.
The overall flow ended up looking like this:
From an end-user perspective, an invitation email allows people to easily opt in to the program by entering their email address. They then receive monthly invitations to participate in specific feature tests or research modules with the option to schedule either remote or in-person sessions. Then, all they have to do is attend the session itself. Everything from the anti-spam opt-in process and self-scheduling to the customizable gifts are designed to give our participants flexibility and control over how often and when they choose to give us feedback.
We’d love to hear from folks tackling similar challenges in user research and design for B2B products. And if you’d like to participate in our User Experience Program, just sign up for a free Segment account and we’ll be in touch— automatically.