Hulu Watch Party

Project Overview

A usability evaluation report on Hulu Watch Party. This is a class project from the department of Human-Centered Design and Engineering (HCDE) at the University of Washington, Seattle.

Project Timeline

Process

Problem: The Hulu Watch Party feature is confusing

During the initial exploration phase, each team member had some barriers in exploring Hulu Watch Party. We wanted to explore other people’s experiences in using Hulu Watch Party. Therefore, we conducted a heuristic evaluation individually and discussed possible usability issues with the interface and user experience.

Then, we designed a usability study plan with the following goals:

1. Assess the overall ease of use and usability of the Watch Party feature for desktop users.

2. Identify obstacles to completing key tasks.

3. Assess user satisfaction with the Watch Party feature for future usability evaluations.

We listed a set of research questions, recruitment criteria, and task lists identifying how to measure task success for our study.

Example Research Questions

• How easily can users find the Watch Party button?

• What obstacles prevent users from starting a Watch Party session?

• What obstacles prevent users from navigating a Watch Party session?

• How well do users understand and use the symbols and icons in the video player?

• How well do users understand the system feedback?

Methods

Recruitment

With clear usability goals and participant criteria in mind, we distributed a screener to recruit target users for the study. The screener includes questions that asked participants’ demographic information, experiences in using Hulu, and experiences in using Hulu Watch Party. The screener was sent through UW student channels and personal networks.

Usability Planning

With the usability goals in mind, we crafted a usability plan that helps us define what successful tasks look like and what data will be collected during the session. We also developed a usability test kit with a detailed timeline, scripts so all sessions can be consistent.

Link to: Usability Plan, Usability Test Kit

We selected a total of 6 participants(1 male, 5 female) who met our criteria for the study. I moderated 2 usability sessions, note-taking 4 sessions, and interacted as dummy participants for 2 sessions.

For each session, there was one moderator, one notetaker, and one teammate serve as a dummy participant. Each participant answered pre-study questions to confirmed eligibility.

During the usability sessions, the participant performed 5 tasks:

• Initiate Watch Party

• Invite a friend to the Watch Party (dummy participant)

• Start the Watch Party

• Interact with your friend(dummy participant)

• Start a new Watch Party

After each task, the participant completed the post-task Likert Scales to rate the ease or difficulty of the task, and at the end performed the System Usability Scale (SUS) questionnaire and answered post-study questions to provide overall thoughts on their experiences and on Hulu Watch Party.

Findings & Recommendations

What went well

• Watch Party icon is recognizable and clearly understood

• Inviting friends to a Watch Party is easy to do

• Sending a message via the chat function is easy to navigate

• Participants felt confident in starting another Watch Party after their initial experience

Opportunities for improvement

We applied Jakob Nielsen’s Severity Ratings to evaluate the issues we discovered from usability testing and rated the areas by severity level. The severity of the issues was measured by their frequency, persistence, and impact on completing tasks.

Finding 1: The watch party feature was not easily discoverable (severity 3)

• All participants expressed frustration while trying to initiate a Watch Party

• All participants had issues finding the Watch Party button

Recommendation

• Highlight the Watch Party feature on the Hulu home page and adding multiple ways to begin a Watch Party (e.g starting within the video player)

Finding 2: The Watch Party synchronization function was unclear (severity 2)

• All participants expressed confusion about why the feature was necessary

• Participants did not understand how the feature decided to sync all of the videos in the Watch Party

Recommendation

• Update feature to sync videos automatically

Finding 3: Messaging feature lacked information on other participants in Watch Party group (severity 2)

• 4 out of 6 participants were concerned with the inability to see the names of guests in their party.

• Participants expressed identification information was important to determine which of their friends was missing from the Watch Party.

Recommendation

• Allow users to choose their display name and show the full name rather than an abbreviated version.

• Provide information on all guests in the Watch Party

Finding 4: Limited interactivity functions in Watch Party (severity 1)

• 4 out of 6 participants could not communicate with their friend (dummy participant) in their preferred method because they are only allowed to send simple text messages and emojis in the Watch Party chatbox.

Recommendation

• Add a voice communication feature and update the chat feature to support more formats like gifs or links.

• Create customized emojis either specific to Hulu or to particular shows or movies to make the experiences more interactive.

Takeaways

A detailed usability guide is necessary for consistent moderating sessions

In the beginning, our team concerned about the consistency in moderating usability sessions with different people. Having a well-written usability guide with clear step-to-step instructions and pilot test helps to ensure all sessions are consistent and ran within the time frames.

Pay attention to the details

We successfully recruited 6 participants for our study and thought the participants eligible for our “important” recruiting criteria, and later found out we miss the environment they use Hulu in. Therefore, some usability sessions did not occur in the participant’s most natural environment and could affect the study outcomes.

Confirm levels of understanding with participants

Participants might have different levels of understanding about follow-up questions. We should always clarify and ensure the participant is answering what we are asking. For example, we conducted the system usability scale (SUS) rating at the end of the usability session and found that participants rated usability by overall experiences instead of initial experiences, which could affect their ratings in the questions.