PROTOTYPE TESTING

Specific improvements decided by user testing findings:

• Sign in with Google and Facebook function added

• Shorten verification page

• Send users confirmation email after completing verification page

• Added progress status dots for onboarding 

• More specific racial categories for users 

• Checklist can collapse and expand 

• Indicator of current time on the timeline 

• The user can manipulate the timeline to show specific programs 

• Scheduling meetings function is on a separate page 

• Add filter to program page 

• Added search bar/filter* options to mentor 

• Added heart to favorite program 

• Changed the presentation of stars on reviews, overall stars are bigger than the review rating stars. (Helps with visual hierarchy) 

• Buzzwords in reviews

• Changed icons to better suit the word they described 

• Made categories more separate with reviews so it’s visually more clear 

• Direct message: got rid of action bubbles

• Added compatibility to mentor snapshot in dm 

• Added edit button to the user profile

• Include mentor onboarding and verification process

• Homepage for mentors 

• Create a review option for mentors

PROTOTYPE EVALUATION ACTIVITIES:

Expert Evaluation: Cognitive Walkthrough/ Think Aloud 

Evaluation Goals: We had several goals for the cognitive walkthrough. We were interested in understanding how users navigated the platform, what users found confusing or unhelpful, and what users valued the most. We wanted to learn more about our users’ experience with the platform so that we could implement better design practices.

DATA ANALYSIS: COGNITIVE WALKTHROUGH

We collected and analyzed the qualitative data from all four expert evaluations for the cognitive walkthroughs. Since we recorded and transcribed the walkthroughs, we assessed our notes from every session and looked for recurring themes or comments.  Afterwards, we compiled a list of significant  findings based on comments, questions, and user suggestions about the platform.

Expert Evaluation: Heuristic Evaluation

We had several goals for the heuristic evaluation. We were interested in obtaining an expert-level assessment of the degree to which we satisfied the Nielsen Norman design principles. Thus, we asked experts to complete a heuristic evaluation where they would rate our compliance with the Neilsen principles and offer recommendations for our platform. We had 4 participants complete a heuristic evaluation based off of Norman’s Usability Heuristics for User Interface Design. Below is a chart of the ratings by heuristic and by expert, on a scale of 1-5 with 5 being the highest.

Target User Task-Based Testing
Our group conducted in-person, moderated think-aloud user testing with target users to evaluate how well our design fits their mental models. We wanted to examine the usability of each function in our prototype, as well as how users’ behaved and their preferences.

Method Strengths  

• Provides us with an understanding of how first-time users would interact with the website

• Provides us with an understanding of first-time users’ preexisting mental models

• Ability for quantitative data analysis in task success/fail rates

• Ability to recognize where issues might occur in real-life application

• Ability to assess issues in user flow

• Ability to observe both behavior and cognition

Method Weaknesses  

• A limitation of this method was that the Figma prototype is limited to the degree to which it acts as a published website, and so we could not collect data for every interaction

• Potential for demand characteristics/biased user feedback (users may often be apologetic for honest feedback or refrain from providing negative feedback)

• Lesser evaluation of design principles, thus coupled with a heuristic evaluation

• A limitation of this method was that it could not evaluate real-life applications such as when in the application process users may actually use an intervention like ours, which may skew the data as we are only testing users once they are aware of the website and its capabilities. In real life, we may see a greater drop-off in users during the onboarding process.

Procedure:

These user tests consisted of three phases: the think-aloud free-explore phase, the think-aloud task-completion phase, and an open-ended interview phase. 3 task-based tests were conducted in-person and 2 were conducted remotely. During the think-aloud free-explore phase, users were asked to use the prototype as if they had opened this website at home and show us how they would proceed in that scenario while speaking out loud their thought process for certain actions or confusions. The free-explore think-aloud phrase would be approximately 10 minutes long. Throughout the entire user test, if users had questions about where certain information was or what specific buttons did, we asked them questions back in order to prompt further action from them such as “how would you find “Help” if we were not here?” 

During the task-completion phase, the interviewer would ask the potential user to carry out tasks on the prototype and again, talk through their thought processes and confusions. During planning, our group decided to prioritize tasks and features that the user had not already encountered during the free-explore phase if we were short on time, but in practice we were able to ask each user every task. The tasks that were asked of participants differed depending on whether they fit the criteria of a potential mentee or a potential mentor. This phase took approximately 20-30 minutes. Responses were measured in quantitative data such as task or success, as well as qualitative data such as documenting cognition.

Finally, during the open-ended interview phase, we asked users about their favorite and least favorite features, as well as other suggestions and concerns. This final phase took approximately 10 minutes. No interview exceeded an hour, with most being completed in approximately 40 minutes.

Data was analyzed through success/fail rates and time taken for tasks for quantitative data and documentation of quotes for qualitative data.

Quotes from Free-Explore Think Aloud:

• “I like how the schools are color coordinated”

• “Application fee and the deadline is good to know”

• “I like how the reviews are divided by categories; in the process, you look at multiple schools for multiple reasons (e.g. affordability); you know where to look if it's divided- I haven’t seen anything like that before”

• “How do I navigate ‘Home’?”

• “Can I add to the checklist if I wanted to?”

• “The onboarding process is clear and simple.

Quotes from Task-Completion Phase:

• “I like how there’s a suggested message already available”
  “It won’t let me choose two semesters?” (onboarding)

• “I have some trouble going back to the program page once I click on the message”

• “I like the schedule page. It is very clear to see all the meeting appointments.”

• “I thought deselecting the checkbox means removing it from the list ”

DESIGN RECOMMENDATIONS FOR FUTURE ITERATIONS + PROTOTYPING