Handbook of Usability Testing

by Jeffrey Rubin and Dana Chisnell

Part 1: Usability Testing: An Overview

Chapter 1: What Makes Something Usable?

  • What do we mean by usable?
  • What makes something less usable?
    • Development focuses on machine or system, not user
    • Target audiences expand and adapt
    • Designing usable products is difficult
    • Team specialists don’t always work in integrated ways
    • Design and implementation don’t always match
  • What makes products more usable?
    • Early focus on users and tasks
    • Evaluation and measurement of product usage
    • Iterative design and testing
    • A multidisciplinary team approach
    • Defined usability goals and objectives
  • What are techniques for building in usability??
    • Ethnographic research
    • Participatory research
    • Focus group research
    • Surveys
    • Walk-throughs
    • Open and closed card sorting
    • Paper prototyping
    • Expert or heuristic evaluations
    • Usability testing
    • Follow-up studies

Chapter 2: What is usability testing?

  • Why test? Goals of testing:
    • Informing design
    • Eliminating design problems and frustrations
    • Improving profitability
  • Basics of the methodology
    • Basic elements of usability testing
    • Limitations of testing

Chapter 3: When should you test?

  • Exploratory or formative study
  • Assessment or summative test
  • Validation or verification test
  • Comparison test
  • Iterative testing throughout the product lifecycle

Chapter 4: Skills for test moderators

  • Who should moderate tests?
    • Human factors specialist
    • Marketing specialist
    • Technical communicator
    • Rotating team members
    • External consultant
  • Characteristics of good test moderator
    • Grounded in the basics of user-centered design
    • Quick learner
    • Instant rapport with participants
    • Excellent memory
    • Good listener
    • Comfortable with ambiguity
    • Flexibility
    • Long attention span
    • Empathic “people person”
    • “Big picture” thinker
    • Good communicator
    • Good organizer and coordinator
  • Getting the most out of your participants
    • Choose the right format
      • Sit-by sessions versus observing from elsewhere
      • “Think-Aloud”
      • Retrospective reveiw
    • Give participants time to work through hinderances
    • Offer appropriate encouragement
  • Troubleshooting typical moderating problems
    • Leadering rather than enabling
    • Too involved with the act of data collection
    • Acting too knowledgable
    • Too rigid with the test plan
    • Not relating well to each participant
    • Jumping to conclusions
  • How to improve your session moderating skills
    • Learn the basic principles of human factors/ergonomics
    • Learn from watching others
    • Watch yourself on tape
    • Work with a mentor
    • Practice moderating sessions
    • Learn to meditate
    • Practice “bare attention”

Part 2: The Process for Conducting a Test

Chapter 5: Develop the test plan

  • Why create a test plan?
  • The parts of a test plan

Chapter 6: Set up a testing environment

  • Decide on location and space
  • Recommended testing environment: minimalist portable lab
  • Gather and check equipment, artifacts, and tools
  • Identify co-researchers, assistants, and observers

Chapter 7: Find and select participants

  • Characterize users
  • Define the criteria for each user group
  • Determine the number of participants to test
  • Write the screening questionnaire
  • Find sources of participants
  • Screen and select participants
  • Schedule and confirm participants

Chapter 8: Prepare test materials

  • Guidelines for observers
  • Orientation script
  • Background questionnaire
  • Data collection tools
  • Nondisclosures, consent forms, and recording waivers
  • Pre-test questionnaires and interviews
  • Prototypes or products to test
  • Task scenarios
  • Optional training materials
  • Post-test questionnaire

Chapter 9: Conduct the test sessions

  • Guidelines for moderating test sessions
  • Checklists for getting ready
  • When to intervene
  • What not to say to participants

Chapter 10: Debrief the participant and observers

  • Why review with partipants and observers?
  • Techniques for review with participants?
  • Where to hold participant debriefing sessions?
  • Basic debriefing guidelines
  • Advanced debriefing guidelines and techniques
  • Reviewing and reaching consensus with observers

Chapter 11: Analyze data and observations

  • Compile data
  • Summarize data
  • Analyze data

Chapter 12: Report findings and recommendations

  • What is a finding?
  • Shape the findings
  • Draft the report
  • Develop recommendations
  • Refine the report format
  • Create a highlights video or presentation

Part 3: Advanced Techniques

Chapter 13: Variations on the basic method

  • Who? Testing with special populations
  • What: Prototypes versus real products
  • How? Techniques for monitored tests
  • Where? Testing outside a lab
  • Self-reporting

Chapter 14: Expanding from Usability Testing to Designing the User Experience

  • Stealth mode: establish value
  • Build on successes
  • Formalize processes and practices
  • Expand UCD throughout the organization