City of Calgary

Understanding how user testing with a diverse sample of end users can improve digital equity outcomes

While volunteering with IncluCity Calgary I participated in my second project, working with a team to conduct usability testing and user research. For this project, IncluCity Calgary and the City of Calgary’s Web and Digital team collaborated on a user testing pilot project for digital equity over the course of 8 months.

ROLE

Assistant Coordinator
Test facilitator
Notetaker
Researcher

TIMELINE

May 2023 – December 2023

TOOLS

Google Meet, Miro

TEAM

14 members

PROBLEM

Digital services often exclude various equity-seeking groups in the design process

There is a digital divide between certain subpopulation groups and the benefits of the City of Calgary’s various digital services. A design process that does not consider the diverse experiences, backgrounds, cultures, or abilities of its prospective users will result in usability issues for all.

Solution

We conducted usability testing sessions with 35 users from 3 different equity-seeking groups on 7 different digital services

By encompassing a more diverse set of end users through including more equity-seeking groups in the design process, more usability issues can be captured. The subpopulation groups of interest for this pilot project were elderly adults, newcomers and new immigrants to Canada, and people with disabilities.
Through user testing with members of these groups, we collected qualitative data to analyze through sorting and affinity mapping to find usability issues for the digital services and traced those issues back to the user groups in which the issues were more prevalent within. The key qualitative metric was the frequency and severity of the identified usability issues.

My contributions

As the assistant coordinator, I worked closely with the project and team leads to set up team meetings, prepare training workshops for volunteers, and coordinate testing sessions with users; I also facilitated and took notes for some testing sessions. Lastly, I coordinated with our client and volunteer team to discuss our analysis methodology and participated in qualitative analysis and synthesis of our testing insights.

Research goals

The City of Calgary established three main goals:

Identify usability issues within various digital services pertaining to subpopulation groups
Demonstrate the effectiveness and value of user testing with equity-deserving groups
Provide a case study to help operationalize and advocate for user testing within digital service development

Process

I worked with team leads to set up and host workshops for the project volunteers on various testing-related topics, such as notetaking and test facilitation. During the testing process, I coordinated 11 user testing sessions, as well as acted as test facilitator and notetaker for 2 other sessions.

The main components of the testing session are as follows:

Initial impressions - users are asked about their prior experiences with City of Calgary digital services, as well as their assessment of whether or not they think the online services are easy to use for someone from their subpopulation group.
Digital service review - users are then asked to complete several tasks within a particular digital service on the City of Calgary website. During that process, users are encouraged to think out loud and share their thoughts and expectations navigating through the digital services.

Analysis

Through close discussions with project volunteers and the UX architect from the City of Calgary, we devised an analysis methodology consisting of two stages:

Organize, consolidate notes by user groups

We first sorted the notes into tables by the user groups, with each row representing a question or subtask, and each column representing a category of observation: general comments, opportunities, pain points, and the successfulness of task. This process was repeated across the data for each of the four testing groups.

Table for organizing user testing notes, with a question for each row and different categories of observations indicated by color

Synthesizing findings into conclusions

Upon organizing the raw notes, we then identified key usability and equity issues by looking for patterns and recurring themes and compiling relevant observations and insights from the raw data to substantiate each claim. The potential subpopulation groups that could be affected by each usability issue were also listed for each issue to further help outline the targeted impact.

Usability issue and user group impacted by the issue, with supporting notes clustered below

Outcome

Upon the conclusion of the qualitative analysis from the IncluCity volunteer team, the City of Calgary’s UX research team used the analysis results to create a report outlining the findings, sharing the report with the product teams responsible for the relevant digital services from testing. The team will soon publish articles talking more about the project and the findings from the report and make them available to the public.

As the report has not been made available to the public yet, I am not authorized to talk about the specific results and findings from the report.

Final Thoughts

This project had the most comprehensive and formal integration of inclusivity compared to my previous project experiences, given that the goal of the project was to understand how focusing on more diverse and underrepresented user groups would yield more valuable results. As a result, it was rewarding to see all the granular works from the months of testing come together in the analysis stage and ultimately condensed into a robust list of key issues.

As the assistant coordinator, I helped with enacting changes and improvements to the testing process and volunteer coordination from learnings throughout the overall project. It would be interesting to see how the City of Calgary's web and digital team builds upon the findings from this project, whether it be conducting more research to drill down to certain groups or refining operational aspects of the user testing process.