The Road to Firefox Monitor

Today we launched Firefox Monitor, a collaboration with Have I Been Pwned, designed to help keep our users safer online. Firefox Monitor is the first of many privacy and security features we’re pushing this year, but it also holds a very special place in my heart for another reason. Monitor represents the fulfillment of the promise we made to our users when we launched the

Shield program. Firefox Monitor was born from a Heartbeat survey and it meandered back and forth through the Shield pipeline several times before it eventually landed in the browser. All that time, you, our users were driving the feature. That’s pretty cool, right?

Space Exploration

We do a lot of exploratory research.

Mark Mayo, Mozilla’s Chief Product Officer, likes to describe this type of work as space exploration. He envisions us as astronauts hurtling through space, launching new satellites into orbit around planet Firefox, intent on discovering new forms of life and planets to occupy. I like that metaphor, too. It makes the work we do sound so perilous and cool.

Like most ideas that come from exploratory research, Firefox Monitor found us and not the other way around. It all started with a Heartbeat survey about Firefox Accounts. We were interested in understanding users’ mental models of Firefox Accounts and trying to figure out where we should be taking Accounts next to provide the most user value. We asked users a broad range of questions about multiple profile use, password management, the ability to “lock” Firefox, syncing, sharing, etc. The real goal was to get a broad sense of what we *could* do with accounts that would be meaningful to users *before* we started building it.

In the first study, we allowed users to vote on multiple items they wanted to see in Firefox Accounts. The list of proposed features was long and we were looking to narrow the scope quickly. We were able to rapidly eliminate most of the list and narrow in on a few strong candidates. One idea performed exceptionally well:

Notify me of potential password compromises

Feature Thunderdome

That’s where things got really interesting. Now that we had narrowed the list of possible features, we needed to find out how they stacked up against each other. We’re still a small company by most standards. We don’t have thousands of engineers on staff standing by to build every crazy idea. In fact, everything we choose to build, means we are choosing not to build/support something else. With these resource constraints we have to do RUTHLESS prioritization

To get a better sense of prioritization, we wrote a Max/Diff survey that forced users to make explicit tradeoffs between features. We took the attributes that ranked highest from the Accounts survey and attributes from a few other exploratory research spikes and pitted them against each other. This is a survey methodology that we employ frequently in our exploratory work. I like to think of it a bit like Thunderdome. In this design, participants are shown a random list of attributes and were allowed to select only one item that they wanted the most and one item that they wanted least. By forcing users to make hard tradeoffs, this design produces the relative user perceived value of each feature as compared to each other. The output of that survey would help provide some directional guidance for prioritizing features across all of Firefox, not just the Accounts space. We knew anecdotally that users cared about privacy and security features. We hadn’t expected for them to rank almost as high as basic web browser functionality:

In that first Max/Diff study, compromised password notifications were ranked almost as high as general web compatibility. Needless to say, we were very surprised. As devout followers of the scientific method we needed to prove that we could reproduce these results. We re-ran this Max/Diff multiple times. First, we used the same attributes list and then attempted to introduce new attributes that might upset the rankings for the perceived value of password notifications. No matter what we did, representative populations of real users told us this feature was extremely valuable.

Making Cupcakes

Our resource constraints don’t just factor into our work in terms of what we ship and do not ship. We hold ourselves to these same constraints for research. We focus on rapid inexpensive tests for validating hypotheses. We try to kill low value ideas early and often. We only continue to invest in more expensive research methods when we have strong evidence to suggest that the idea has high potential. Internally, we refer to this as making cupcakes instead of full-sized cakes

At this point, we had reached the limits of what we could do through surveys and attitudinal research. We knew that users believed (or felt that they *should* believe) that the idea of password notifications provided exceptional value. I felt that I had enough evidence to justify a more expensive, but ultimately more definitive, test to validate these findings.

The “cupcake” version of Firefox Monitor was called Security Advisor at the time and was one of the first Shield Studies we shipped. Everything we do is a value exchange. We needed to understand if this feature was valuable enough for users to give something back in return. This would be the clearest signal yet that we should build this service for real. In that first Shield Study, we generated a static list of websites that had recently been hacked and stored them locally on the user’s computer. If they visited a website that had recently been hacked, we intercepted them with a drop-down notification that provided information on the security breach and steps to address the security concern. We had great interaction with these initial notifications.

The real test was what happened next. After the user had time to take action on the breach, we asked them if they would be willing to sign up for a Firefox Account if that meant they would receive instant notifications of future breaches. The design wasn’t pretty, but it didn’t have to be.

The attempted sign-up rate was higher than we’d seen for any other Accounts offer to date. The service didn’t actually exist since this was just a dry market test and we ended the experiment there. We notified users that this had only been an experiment and asked them to provide additional feedback in a survey. This experiment told us a great deal. Two things in particular stood out to me:

1. The value exchange is high enough that a high proportion of users are willing to go through the process of creating an account to get access to the feature.
2. In the debrief surveys after the experiment, we saw a high degree of self report that a feature like Security Advisor made participants feel safer online.

I was convinced. My teammates were convinced. We shared the results with Peter Dolanjski, Product Lead for Firefox. He was definitely convinced. It was exciting! We’d let our users drive the feature development process and landed on something exceptional and very much aligned with our mission and values. Peter took the next step of setting up a meeting with Troy Hunt, the creator of Have I Been Pwned.

At the time, I knew Troy by reputation only. I’d used his service myself. Shortly after that first experiment, I had the pleasure of meeting him in person. It’s always scary when you meet someone that you admire a great deal for the first time. Troy didn’t disappoint. His passion and genuine concern for the safety of online users was contagious. I left that meeting feeling incredibly hopeful for a future collaboration.

My team builds prototypes. We build things that are quick and cheap. Our work is throwaway work, not the elegant and polished work you see released in Firefox. It was at that point that we handed off this project to the “real” engineers on our Security Engineering team. Our UX, UR, and Content Team gave the project a proper name and gave the experience depth and character. They’ve done an amazing job turning the rough chunk of stone we gave them into the fully fleshed out product you see today.

The Road Ahead

The product we shipped today isn’t the end of the road for Firefox Monitor. This is just an MVP. We aren’t done iterating and we probably won’t ever be. I’ve already had the pleasure of working with the Security Engineering team on two more Shield Studies for Firefox Monitor, all before we even launched. Expect to see Monitor growing and showing up in more places in the Firefox planetary system soon.

While Monitor is the first feature exclusively from the Shield program to land in Firefox, it won’t be the last. You’ll be hearing a lot more about how Heartbeat and Shield Studies are putting users back in charge of Firefox over the next few months. We have some amazing things in store for you. It’s going to be a great year.