The (Truly) Complete Guide To Usability Testing
No matter what your specific role is on the product team, chances are that you’ve heard of usability testing. There’s a good reason for that—usability testing is quite possibly one of the best, cheapest, and most accurate ways to get insights about the user experience that ultimately leads to you meeting your goals and KPIs. As a user research lead, I live and breathe usability testing and whether you’re new to it or just want to brush up on your skills, you’ll find what you need in this handy guide.
Why Is Usability Testing Important?
If your whole team isn’t already convinced, it’s worth taking the time to educate and communicate with others about the utility of usability testing in the product design process so that you’ll have buy-in when you have user-based product iterations that you want to move forward with based on your usability insights. Here are some of the main reasons that usability testing is almost always a win.
Save time on internal debates
Unless your organization is the exception to the rule, it’s likely that everyone and their mother has an opinion about the user experience of whatever feature or flow you’re working on. It’s probably also true that to an extent, most people have an informed opinion based on industry knowledge, product insights, and so on.
The problem is that you could debate forever, waste a lot of time, and still not be aligned when you release a compromise of sorts. With usability testing, you test your feature or flow with actual relevant users so that you can make UX decisions that are based on reality rather than the assumptions of your team.
Usability testing offers insights no matter where you are in the product development process
If you’re pre-release, your usability testing in the early stages will make sure that your initial release is based on the reality of your user base. If you’ve already released and you’re waiting for quantitative performance data, doing usability testing while you wait can help you be ready with ideas for iterations to improve whatever you see in your metrics. If you’re post-release, you can get user-based insights to help you prioritize and decide on your next iterations.
Usability testing offers the insights you need whenever you need them.
Usability testing helps you avoid bias in decision-making.
Whether it’s because someone on your team is reluctant to disagree with leadership or the person championing the feature feels 100% sure that they know exactly what to implement and how—every product team runs the risk of internal issues leading to biases in decision-making.
With usability testing done well, you’ll end up with insights and direction that bypass all of these potential biases so that what you release to users is based on their needs and experience and nothing else.
Who Should Own Usability Testing On Your Team?
While there’s no hard and fast answer as to who should conduct usability testing, here are some things to consider when deciding on ownership and execution of usability testing:
- If you have a UX research team, this is most likely the best place to start. The researchers on your team will know best practices and have familiarity with relevant platforms to get the job done fast, but also with just the right amount of rigor.
- Designers on your team may also have the right skills. Amazingly, not all design education includes research methodologies like usability testing—but many designers learn this on the job. If you don’t have a UXR team, check in with your designers.
- Other product stakeholders can learn. If you’re leading this initiative and you’re, say, a product manager—usability testing does require some knowledge, but if you read through this guide and perhaps consider a short course, you could certainly master the skill.
Types Of Usability Testing
Usability testing in general involves testing a feature or flow with members of your target audience and/or your current user base. However, there are several different types of usability testing that each give different types of information. Here’s a brief overview of specific types of usability testing and when they’re useful:
Moderated usability testing
Moderated usability testing is when you or another moderator conducts your test with users in real-time, either in-person or remotely on a video call. The primary value of moderated testing is that it allows you to ask questions and clarify things with users as they’re completing the test. It also allows the flexibility of asking more interview types of questions regarding user needs, behaviors, and motivations.
The downside of moderated usability testing is that it’s quite time-consuming: not only do you have to be there for each session, in-person or on a video call, but you also have to schedule with users and handle all of the relevant admin.
I generally recommend this type of face-to-face or remote usability testing only when the feature or flow is very high stakes—it’s not a relatively minor iteration—and when you’re still questioning things like value to the user and so on.
Unmoderated usability testing
Unmoderated usability testing is when you send relevant participants, usually via a platform like my personal favorite UserTesting.com, to conduct a test that you’ve written in advance. There are a variety of usability testing tools, and a quick Google search will churn out a bunch of great options. This is always within the category of remote usability testing. You receive the recorded sessions instead of being live with the users as they do your test, and then you can watch and analyze the sessions on your own schedule.
The primary advantage of unmoderated testing is that it’s a lot faster: you write the screener to recruit participants on whichever platform you use, you write the test and relevant questionnaires, and you wait for the recordings to come in. You can analyze them on whatever schedule you want, and so if you’re pressed for time, you can turn it around fast and share results with your team.
The downside of unmoderated testing is that you can’t ask your testers questions as they go through each session, which can feel limiting, particularly if they do or say something that surprises you.
Simple usability testing
Whether you choose to go with a moderated or unmoderated testing process, most product teams utilize the simplest and cleanest form of testing at some point, which involves giving users a series of tasks to take them through a flow or utilization of a feature. As users complete tasks step by step, they often enlighten us to UX or even functionality issues at every stage of a flow.
In this type of testing, you get the benefit of uncovering usability problems from real users as they go through the flow.
Five-second testing
Five-second testing is a method whereby you show users a landing page or home screen for five seconds and then ask them to share their impressions and what they noticed.
This methodology is most appropriate when your team is looking for first impressions that are as close to the new user experience as possible. It does not explore user attitudes toward functionality or look at the user experience in-depth when it comes to features and flows.
If these are the types of insights that your team is looking for, the primary advantage is the fact that this type of testing session can be done quickly and remotely. Additionally, the moderator can generally be anyone on your team who is a good listener and knows the basics of asking good questions.
Card Sorting
Card sorting, while very specific, is an extremely powerful methodology if it aligns with your research goals. In card sorting tests, users are given cards with words on them and asked to group them and/or order them according to what makes the most sense for them. A practical application of this, for example, is deciding how to group features in a toolbar. By giving feature names to users on cards and asking them to group them intuitively, you can get a sense of how and why users would intuitively expect to find certain tools.
Although this can be done in an unmoderated setting, I generally recommend that these tests be run by a moderator. This is because how users group your cards is insightful, but the ability to ask as many questions as you need, in real-time, about why users make certain grouping or ordering decisions offers much richer insights.
Card sorting doesn’t touch user experience in a direct way, so make sure that information architecture is your goal before you choose it as a method.
First-click testing
First-click testing is both a technique and a test in and of itself. What happens is that the moderator or testing instructions direct a user to complete a task, and what’s recorded or noticed is the first place that a user clicks in order to accomplish the given task. This helps you check whether your user experience is intuitive for a specific task.
Often, when doing any usability test that involves giving users multiple tasks, the first click is an important metric. If your research goals go beyond where a user would first go to complete a task, first-click testing is still a useful method and metric to keep track of, even if you’re doing a more robust usability test.
Navigation Testing
Navigation testing, similar to first-click testing, can be a test within itself or a part of a more detailed usability test. Users are given a task or goal and asked to complete the task within your product. Here, you’re noticing how users are navigating in order to complete various tasks and learning whether or not the information architecture within your product is intuitive for users with certain goals or outcomes.
If you’re only concerned with information architecture and whether or not it’s user-friendly, it’s definitely possible to design a navigation test in isolation and get a lot of value out of it.
Preference Testing
Simply put, preference testing gives users different variations of the same thing—perhaps different designs or even functionality—and asks for them to indicate and explain their preferences.
This type of testing can be useful, but not necessarily for the reason that you think. Often, preferences are a shaky metric—people might be more impressed by a design or feature at the moment, but none of us are experts at predicting future behavior. In other words, what people prefer now may not have staying power or be relevant outside of a testing environment.
So why do it anyway? The trick with preference testing is to pay the most attention to the reasons behind user preferences. When you do this type of testing, you don’t want to go with the option just because most people that you tested had a preference for it. You want to learn more about how people articulate their needs and challenges as it relates to the product tasks ahead of you. Then, you can use these more in-depth insights to help make actual product decisions.
Preference testing can easily be done either in person or remotely; doing it remotely will likely save some time and effort.
Tree Testing
Tree testing is a way to test the discoverability of specific topics or features on your website or product. It’s different from navigation testing because rather than being given a more complex task, test participants are shown only the navigation tools (such as a website header or an app toolbar) and asked to find specific topics or categories.
Tree testing, often called reverse card sorting, can be done remotely or face-to-face. Either way, your follow-up questions—for example, why users looked for a certain topic under a certain category on your website—will be rich with insights about how information is organized and whether or not it’s user-friendly.
What about a usability lab?
While you should be convinced by now that usability testing doesn’t require anything fancy, if you’re lucky enough to have access to a usability lab, you’ll open up a whole lot of fun methodology—eye tracking, utilization of physical products, and so on.
Even if you don’t have access to a lab, check this out just for fun.
Usability Testing: Tips And Best Practices
Planning and writing your test
Whichever method you choose, exactly how you write your test will have a huge impact on the quality of the insights that you get from usability testing. Here are a few things to keep in mind as you’re getting started.
- Write out your research goals: What exactly do you want to learn from your usability test? Having this as a guiding principle at the top of your document as you write your test will help make sure that you’re asking the right questions that suit your goals.
- Speak how your users speak: Most of us—guilty as charged!—use a lot of product speak in our day-to-day work. Make sure that when you’re writing instructions or tasks, you use the words that your users use so that they know exactly what to do.
- Make sure that each usability task only has one action: A lot of times, we’re tempted to tell test participants to take multiple steps in one task. The problem is that when we do this, it can be information overload, and testers often miss steps or nuances. In the end, this makes some tests shaky in their reliability and, in some cases, even useless.
- Have a colleague look over your test before you put it in front of users: Often, my teammates will find things in my usability tests that I thought were clear but, alas, were not. Having someone look over your test helps ensure clarity.
Recruiting participants
How and who you recruit to participate in your usability tests is a huge factor in the reliability of your insights. After all, if I’m building a photo editing app for Gen Z—how relevant are my mom’s usability issues?
You can use the Nielsen Norman guide for choosing the number of participants needed for your usability test. Once you have your sample size in mind, use the following tips to guide you:
- On testing platforms, users sometimes lie in order to qualify—take steps to prevent that: You can count on everyone taking your screener to be truthful when money is involved, unfortunately. People may try and understand what you want to hear and answer accordingly. There are a number of strategies for making sure that you weed out people who are not being truthful in screener surveys, which you can read about here.
- Consider having more than one user group: Often, you’ll find that different groups of users will yield different insights—and they’re all useful. For example, I often test with users who already use our products in addition to users who have never tried our products. That gives me insights into how both existing and new users experience whatever we’re working on.
- Always remind users of upcoming sessions: If you’re doing any type of moderated usability testing, whether in person or remotely, make sure that you remind participants of upcoming sessions and ask for confirmation of their attendance. Missed sessions mean wasted time for you.
Analyzing your testing sessions
There is a common misconception that you can just watch testing sessions or review your notes from live sessions and get a sense of your test results. Unfortunately, it’s a little more complicated than that. Just getting a senseI leaves things up to your interpretation, which can be rife with user research biases.
Follow these best practices to make sure that you’re interpreting your usability test results properly:
- Use a simple but scrupulous analysis method: For example, affinity diagramming is a simple but useful way to extract themes from any type of qualitative data.
- Analyze ALL of your sessions: It can be super tempting to analyze a few sessions and call it a day, especially if you’re seeing similar results. The problem with this is that even if users have similar experiences, the nuances in their testing sessions often provide very useful insights. Not only that, but anyone who has done significant usability testing knows that sometimes, the first five sessions yield one insight—and the last 5 churn out something completely different!
Happy Testing!
There’s no question that any and every product team can benefit from some type of usability testing that gives deep insights into user behavior and provides real-time user feedback. Within the realm of usability testing, there are research methods for every goal that can help you identify usability issues that ultimately enable you to better serve your users.
Now that you’ve educated yourself with this guide share it with team members and motivate them to run usability tests for the entire organization’s benefit! If you’re still feeling overwhelmed, you can check out some usability testing templates or tutorials for further inspiration.
Also, if you’re utilizing usability testing methods of any kind, be sure to subscribe to The Product Manager’s newsletter!