Conducting Usability Testing: Maximizing user experience

Have you ever found yourself scrolling through an app, trying to accomplish a simple task, but finding yourself confused and frustrated? Most of the time, as users of a product, we tend to blame ourselves when we are unable to complete certain tasks while using a product. However, the issue might lie with the product itself and not our ability to navigate it. As designers, we are responsible for ensuring that the products we design and deliver address the needs of users, helping them achieve their goals efficiently and effectively.

What is usability testing?

Usability testing is an essential part of the Design Thinking process, aimed at assessing the effectiveness of a product or service.

By discussing with prospective or existing users, we can determine how well a product aligns with their objectives and facilitates task completion.

Types of usability testing

The usability testing method you choose depends on the needs of the project, the budget and the time you have available.

Conducting usability testing

Before conducting the sessions, we need to ensure that we cover all the necessary points so that the data we gather is valuable and can be used to inform future design decisions. Here is a possible structure for the Usability Test Plan.

You can use Miro, Word, or any tools that help you organize information.

Overview

• Define the test objectives – What is it you want to determine by running these sessions? Are you evaluating the ease of use of the product for new users? Are you assessing the clarity and effectiveness of messaging and labels?
• Introduce the methodology – Think about the number of participants, the tools you will use, and whether the sessions will be held remotely or not.
• Include information about the participants – How many people will you conduct the sessions with? What is their background? How did you recruit them?
• Contour the structure of the test in general terms and introduce the people involved – Who will be the moderator, and who will take notes? Are there going to be other people joining the sessions as observers?

Define the usability tasks & goals.

• Describe the scenario that is going to be followed during the usability testing sessions, as well as the tasks and the objectives they address. Remember that the tasks must be realistic, so think about activities that could actually be performed by people when using the app in a real environment. Exclude tasks that are fictional and highly improbable ever to be performed using the product you’re testing.
• Introduce the usability goals that the research will be focusing on (e.g., completion rate, error-free rate, time on task).

Define the scenario and questions that will be asked

• Going through the scenario step by step and presenting the questions that will be asked (or that could be asked, depending on the situation), ensures that all the stakeholders involved in the process are aware of how the relevant data has been gathered.
• Include the list of pre-test and post-test questions as well.

Define the problem's severity and frequency.

• Each issue you identify will be classified depending on severity and frequency and will be translated into an actionable item to be addressed by the team involved.

Severity

• High – prevents the user from completing the task
• Medium – causes difficulties for the user when trying to complete the task
• Low – represented by minor problems that don’t affect the users too much

Frequency

• High – 30% or more users experience this issue
• Medium – 11-29% of users experience this issue
• Low – 10% or fewer of users experience this issue

Running a usability test session

As recommended by Jakob Nielsen, we can conduct usability test sessions with five users per segment, with this number being dependent on the segments we are testing. During these sessions, several roles are taken by members of the team:

• Facilitator
• Data logger
• Test observers
• Test participants

Here are the main steps included in the process of conducting usability testing:

• The facilitator is the one who starts by explaining the purpose of the research to the participant, as well as describing the way the sessions will go.
• Having signed an agreement to participate, the participant and facilitator start the sessions.
• The facilitator asks some intro questions before going through the scenario prepared.
• Having gone through the entire scenario, the facilitator asks several questions to address the usability goals and to calculate the SUS score.

💡 What is the SUS score? The System Usability Scale score is a quick and reliable way to assess the usability of a design. It measures perceived ease of use, suitable for a wide range of digital products and services. It provides an overall usability score based on ISO 9241-11, focusing on the following characteristics: effectiveness, efficiency, and satisfaction.

Best practices when running a usability test

There are several things we should take into consideration when we conduct these sessions, to ensure that everyone involved is comfortable and that the sessions result in the most valuable data.

• Ensure that the objectives you define are clear.
• Recruit participants that represent your target audience.
• Prepare a test plan before running the sessions, and discuss with the people involved in the process based on this to make sure you include everything that is needed for successful research.
• Create realistic scenarios so that the results reflect real data. The tasks that you include must be activities that people would normally do when interacting with the product you are testing.
• Maintain a neutral approach and avoid leading questions so that the data you collect is not biased.
• Don’t ask people what they like, but rather find out how your product can bring them the most value when conducting certain tasks.
• Transform the findings into actionable items, iterating along the way so that the product you design is improved based on the data collected.

Putting usability testing into action: a project spotlight

The best way of learning is through practice, so let’s have a look at a real-life example of how usability testing can help us uncover insights into how our products behave and how we can improve them.

We conducted usability testing on one of the websites we designed (craft.centric.eu) so that we could analyze the degree to which it helps people achieve their goals. Here are the steps we followed to gather the data:

Tools used:

• Miro (for planning the test and visualizing and analyzing the results)
• Microsoft Teams (for conducting the tests remotely)
• Word (for creating the usability test plan and the results report

The usability test objectives focused on several areas such as:

• Whether or not the navigation is intuitive enough.
• Determining to which degree the website helps people achieve their goals.
• Analyzing if the website causes any kind of frustration to users.
• Determining if there are any inconsistencies in the interface and experience.
• Analyzing user satisfaction with the website.

Having conducted the sessions and analyzing the data, we put together a results report that acted as a guide for us when improving the product. Whether we uncovered features that we could implement, bugs that we could fix, or improvements that we could make, all of these encompass a complete and successful experience for our users which brings value and helps them achieve their goals.

So, let’s have a look at some of the results!

The first information we present in our report is an overview of the results, including the speed, accuracy, overall success and satisfaction of people performing the tasks. We also include the SUS score calculated, to ensure that all the important information is available to people reading the report right from the start. We follow this with a list of the tasks, with the results for each.

What is also important to present in the report are the practical recommendations for improvement. We chose to present them as below, showing a screenshot of the screen/section we analyzed, the feedback from the usability test and our improvement recommendations. You can also add here a section about what these improvements translate into from a business perspective – will a certain change in the design increase user productivity? Will more people subscribe to the newsletter?

After analyzing and putting together the results of the usability test sessions, we were able to back up our design decisions when discussing with stakeholders, as well as emphasize the importance of improving certain areas in order to create a product that brings value to users. Here are some common usability testing questions you can include in your research, as well as some you should avoid:

• Can you name any competitors that you currently use?
• What features do you find most valuable and why?
• Are there any features you find to be least valuable? Why?
• I noticed you [describe something they did]. Why?
• How was your experience completing X task?
• What do you expect X product/feature to be like in the future?
• If you could make any changes to X, what would they be and why?
• What’s the most important thing on this screen for you?
• Without clicking on it, what kind of page would you expect to find on the other side? What would it contain? How would it look? Without clicking on it, what kind of page would you expect to find on the other side? What would it contain? How would it look?
• Do you have any immediate comments on the website?

Questions to avoid:

• Do you like this product?
• Do you like this feature?
• What do you want?
• Do you like this feature better than the other one?

Challenges and how to overcome them

As crucial as usability testing is in the design process, it can also pose a few challenges, some of which we’ll cover below.

• Recruiting participants: As mentioned previously, it’s essential that we conduct these usability test sessions with representatives of our target audience. Recruiting these people can sometimes be challenging, but a reliable way to overcome this is to create define clear user personas based on real data.
• Removing bias: Usability test sessions are usually conducted by a moderator, a note taker, and observers, the number of which depends on each situation and project. This being said, given the number of people present, participants’ behavior can be influenced (this is why we never ask people if they like something, as they will most likely say they do even though they don’t just to not risk “upsetting” the people involved). To address this challenge, you want to make sure that people feel as comfortable as possible, and that they are always aware that it’s not them that is being tested, but rather that they are testing the product. It’s also important to avoid leading questions, which can be done by using open-ended questions, as well as neutral language.
• Reduced participant engagement: When it comes to usability testing, you will conduct these sessions with different people from different backgrounds. Some of them might know exactly what they are supposed to do, while others might be nervous and refrain from sharing some of their feedback. To overcome this, ensure that the environment is comfortable (as useful as usability test labs might be, they can also be quite overwhelming for some people as they are very formal), and that people are made aware from the beginning of how the sessions will be conducted. Use of probing questions (e.g. “Can you explain why you choose to click on this button to achieve your goal?”, “Can you describe any challenges you encountered during this task?”) could help overcome this challenge.

Conclusions

Conducting usability testing is an essential step when designing products that address people's needs and help them achieve their goals efficiently. While the end users are the main point of interest of this study, the process benefits multiple stakeholders involved, such as designers, product managers, and development teams. In the end, usability testing comes in handy for teams so that they maximize the value delivered to users.

References

https://www.interaction-design.org/literature/topics/usability-testing

https://www.interaction-design.org/courses/conducting-usability-testing

https://www.nngroup.com/articles/task-scenarios-usability-testing/

https://careerfoundry.com/en/blog/ux-design/how-to-write-usability-testing-questions/

https://xd.adobe.com/ideas/process/user-testing/sus-system-usability-scale-ux/

https://www.usability.gov/how-to-and-tools/methods/usability-testing.html

https://www.nngroup.com/articles/usability-testing-101/

https://maze.co/guides/usability-testing/

https://www.hotjar.com/usability-testing/methods/

https://www.nngroup.com/articles/usability-testing-101/

https://www.productplan.com/glossary/usability-testing/

https://usabilityhub.com/blog/usability-testing-questions