Paper Prototypes & User Testing

Due: Tuesday, Nov 1 11:59PM

Assignment Overview
Your assignment is to convert a subset of your more promising sketches from the last assignment into paper prototypes and to test them with four users. You will create and test two different versions of your prototypes. As I've noted before in class, comparing two designs is far easier/better than evaluating one single design in isolation. For example, it's easier for a user to express preference between one design over another rather than articulating, exactly, why they don't like a single design.

Following from the IRB protocols we discussed in class, your participants must sign an informed consent document before taking part in your test. With the participant's permission, you should video record all sessions. This can be done with any type of recording device from a DSLR to a mobile phone camera.

The video recordings will be used to create a video report (you will also use them later in the semester for the last team assignment, so do not lose them!).

What to Do

  1. To start, watch the Nielsen Norman Paper Prototyping: A How-To Video. This will help prepare you for this project assignment and the next one as well. In addition, prototyping--and, specifically, rapid prototyping--is perhaps one of the most valuable techniques in HCI/design. So, it's worth spending some time on the readings and, of course, applying the prototyping concepts in your projects.

  2. Iterate and refine the three primary tasks that users should be able to accomplish with your application (based on learnings/reflections from the last assignment).

  3. Then, transition to the first core part of the assignment: riff, iterate, and create two different paper prototypes for the three primary tasks. That is, you must create "Paper Prototype #1" that allows your users to accomplish the three tasks one way and "Paper Prototype #2" that allows your users to accomplish the three primary tasks another way. The paper prototypes should be functionally different so that your users can compare and contrast their experiences with both of them. So, for example, the two prototypes should represent the tasks in fundamentally different ways. Remember, the focus here is not on aesthetics/beauty but rather on understandability, usability, approachability, and, to some degree, layout, widget type, etc.

  4. Once, you've created the two paper prototype designs, beta test them with members of your team and make requisite changes. This "eating your own dog food" is a good way to catch errors before investing time in testing with actual users.

  5. Now you're ready for real user testing (the second core part of the assignment). Recruit four independent users to test out your paper prototypes. Now, ideally, these users should be members of your target user group(s); however, for the purposes of this class, the users can be anyone (including from this class). Each user testing session must be done in isolation (that is, you cannot have more than one user testing at a time) with at least two experimenters present. For the testing session, you should follow this protocol:
    1. First, download and modify this IRB "informed consent" template to fit your project [link]. At the beginning of the user testing session, read the "Purpose of this Study" section of the consent form out loud to your participants. This should be done consistently for each participant. Then, give your participants a chance to read the entire consent form, ask questions, and, if they agree to participate, have them sign the form. If they do not agree to participate, simply wish them a nice day and recruit another participant (it can be slightly awkward but this happens!). If they do agree to participate, provide a copy of the form and take the signed copy for yourself (please scan in the signed consent form and include it in your report appendix).

    2. After the informed consent process, you can begin user testing the prototypes. Follow the method described in the Nielsen Norman "Paper Prototyping: A How-To Video." Because you have two different prototype designs for your tasks, you should fully test one prototype for each of the three tasks before testing the other prototype. Because all subjects are exposed to all conditions, this is a "within subjects" study design. Consequently, you need to counterbalance your prototype conditions. In this case, two users should test Prototype #1 then Prototype #2 and the other two users should test Prototype #2 then Prototype #1. The condition order for all four participants should be set before you test your first participant.

    3. As the user is interacting with your paper prototypes, ask them to "think-aloud." One experimenter should record notes about how the participant was using the prototype, the problems/successes encountered, and comments made during the testing sessions. Scan in and include these raw notes in your appendix for each user testing session. I would like each individual student to serve at least once as a "observer/recorder" and at least once as the experimenter running the session.

    4. After both prototypes have been evaluated, provide your participant with a short paper survey with a few questions asking about which prototype the user liked better and why as well as questions seeking participant ideas for improvement. Scan in and include these responses in your appendix for each user testing session.

  6. After user testing, analyze your collected data (e.g., observational notes, post- survey data). As you have only four users, it does not make sense to perform statistical analyses but you should report the raw data (e.g., in tables or graphs), which should indicate trends (or the lack thereof).

  7. Finally, you need to create a video report. See below.

The Video Report

  1. Title screen: app title and team name
  2. Intro: quick overview of your application and three tasks
  3. Prototypes: introduce the two different paper prototypes and clearly demonstrate each of the three tasks with both prototypes
  4. Pilot testing: explain what you learned from pilot testing with your own group members and how you changed your paper prototypes and/or your testing protocol for the actual user studies
  5. User testing method: describe how you recruited your participants, their demographics (if relevant), where you performed the user tests, how long each session took, the data that you collected, and how you analyzed the data
  6. User testing results: describe key findings and user patterns that you observed during the study
  7. Potential paths forward: describe some implications for design based on your user study (these are things that you will work on in TA05)

Formatting Guidelines

  • Videos should be 7 minutes or less
  • Use on-screen annotations and titles to help guide the viewer and break up sections
  • Your video should have some spice/pizzazz!

Appendix

The appendix should include:
  • A scan/pictures of all paper prototypes created for the two designs that cover the three primary tasks
  • A scan of the four signed informed consent forms
  • Raw notes from four user testing sessions (notes should be clearly marked with a timestamp and session number)
  • A scan of the post-study paper survey responses (these should also have session numbers)

Rubric
You can download the rubric as an Excel file below:


We've also taken a screenshot of the rubric for your convenience. You can click on the image below to enlarge.
TA04-PaperPrototypes-GradingRubric_v1_MLM_Cropped_650w.png