Usability Study- Alexa Integration with Zoro
For my third project, I collaborated with the software engineering interns who were working on a very exciting project of integrating Zoro.com with Alexa. After the first iteration of development, we wanted to study the users shopping behavior of using the developed Alexa voice commands by observing them in action and understand how they perform the tasks and achieve the intended goals.
We conducted moderated task completion test with Zoro.com employees as potential customers. We selected employees of Zoro.com since they have a working knowledge of the product, an interest in using voice interfaces, display diverse online shopping behavior and limitations with resources and time for the completion of the project. As a result we identified ways to make the product more functional, usable and desirable.
For this prototype/ concept testing, video and audio recordings were collected on all the interviews for the purposes of data analysis. We conducted a pre and post survey. Individuals participating in the test was asked to perform a set of tasks and interact with the prototype through voice commands and we asked them to think out loud about their thoughts, feelings and expectations. The test for each participant lasted for about 30 min.
Study Goals
1. To test Alexa integration with Zoro.com for users to search for items, receive the search results via email, and select items through voice commands.
2. To identify if users are able to understand the responses from Alexa.
3. To identify if the users are clear about what voice commands they should input next.
4. To identify the learn-ability of using Zoro.com through Alexa.
5. To identify usability issues with the responses in the form of voice, email or text.
6. To identify sources of errors.
7. To identify what functionality and response the user expects from a vocal experience of using Zoro.com
8. To measure if users are able to complete what they intend to do.
9. To measure the ease of use of using voice interface for shopping in Zoro.com.
Process
Planning
Usability Study
Analysis
Reiteration
The existing code and commands are modified based on the insights
We thoroughly analysed the data we collected and discovered insights.
In this phase, we conducted surveys and executed the usability study of 11 participants through the span of 2 days.
We planned the logistics and execution of the usability test.
Recruitment of Participants
We chose 11 participants from Zoro that matched our requirements based on even distribution of age group and expertise in using voice interfaces. We planned to execute this on a span of two days in two Zoro locations.
Pre survey
The pre-survey was to understand participants comfort level in using voice interfaces and what their main contexts and purposes of ordinary usage are. We also tried to get insights on their past online shopping experiences.
Execution of the usability study
We provided a list of selected tasks for the participants to perform using Amazon Echo Dot. All the tasks were created for specific inquiry of a function that Zoro performs. We identified the functions and broke down the high-level tasks to sub-tasks for which we created a task sheet consisting of the task, the emotion the participant experiences, the feeling of completion and comments if any.
We took notes on our observation for each participant and compared against the completed sheet. After completion we interviewed them briefly about their experience and to gather feedback.
We measured the responses with the metrics -
1. How confident a user is with interacting through the voice interface
2. How easily a user can interact through the voice interface
3. Satisfaction level when interacting through the voice interface
4. The number of times the user has to repeat the voice command
5. The number of times Alexa gives an unexpected response
6. The amount of time it takes the participant to complete the task
7. Successful task completion
8. Critical errors that stop the participant from test completion.
9. Indirect completion of tasks.
Desirable Outcomes
1. A user says the correct utterances to invoke specific intents.
2. A user is able to complete all tasks successfully.
3. A user is able to identify when a task is completed.
4. A user is satisfied with the tasks they are performing.
Post survey
In the post survey, we inquired if any change in comfort level was experienced in their interaction. We also examined their confidence, frustration, satisfaction, difficulties when completing the tasks.
Analysis and Insights
.png)
.png)
.png)
.png)
.png)
.png)
.png)
.png)
.png)
.png)
.gif)
