Q&A on User Testing
Most product managers recognize the value of user testing. But sadly, we don’t often allocate the time to do it properly on new feature/product ideas. We are all under intense time pressures. So it is often hard to fit in testing and design iteration before the sprint deadline.
Steven Cohn, the Founder of Validately and a user testing expert, gives some practical advice on how to balance the time pressures of product management and conducting proper user testing.
(SJ note: I am an advisor to Validately.)
SJ: Many executives think it’s the product manager’s job to know what customers want. Why should a product team allocate its limited time to user testing?
SC: I hear that all the time… I call it the “Steve Jobs” effect. But most people have the wrong perception of Apple. If you talk to someone familiar with Apple’s approach, you will learn that Apple does more user research and testing than almost any company in the world. They just do it in secrecy, so we assume Apple’s designers are magicians.
Let me hit you with some industry stats:
- 50% of the features in a typical product are unused by its customers. That’s right; half of what you are designing and building right now will not be used by customers.
- 30% of a typical development team’s efforts are classified as “re-work.” Meaning, re-coding an existing feature to better reflect customer requirements or to make it more usable.
Those are massive numbers. Real needle moving stats and it is widespread. The truth is, it is impossible for a Designer or PM to sit in a room and guess how a customer wants to use a feature/product.
SJ: What are the different testing options? and why and when should you do each type of testing?
SC: I will explain user testing using this grid:
| Moderated | Un-Moderated | |
| UI Usability | Yes | Yes |
| Real Life Usability | Yes | No |
When you are testing a new feature design, you want to learn 2 things:
- UI Usability
- Real Life Usability
UI Usability is easy to understand. Can someone go from page 1 to page 2 to page 3 to page 4 without getting confused. Real Life Usability is more complicated. This test type answers whether a user will actually use the feature in their real life. As simple as it sounds, there is a lot that goes into that answer.
Remember this one thing when testing Real Life Usability... What a customer says they will do, what a customer says they did and what a customer actually does are three different things. [Tweet this]
Example questions when testing real life usability are:
- How they will use the feature in their daily life?
- How frequently?
- In which occasions?
- What do they view as alternatives to that feature?
- What do they like about that alternative?
- What do they hate about that alternative?
- What will make them switch from that alternative?
I can go on and on about the questions that help predict the success of a feature design. But the big point is this... until you put a prototype of a new feature in front of a customer, there is NO WAY to determine whether it meets the customer’s requirements to actually use it. Simply gathering “requirements” in interviews is not good enough. Customers have to see something visually to tell you if they would use it.
With this basic understanding of the goals of user testing, the next part is how.
Moderated Testing means testing with a customer in real time. It can be remote or in person, but you are live with the customer asking them to try a prototype or already built feature and asking them Real Life Usability questions (noted above). You can also learn UI Usability with Moderated testing.
- Benefit:
- The benefit of Moderated testing is that you learn deep insights on how a customer will use a feature design. You can follow up and ask questions when a user mentions something and dig in deep.
- Tradeoff:
- This testing type is a bigger time commitment.
- How to set up this test:
- This is best done using a prototype because it is easy to make changes
- Recruit testers that meet your persona & schedule a time to do a live session. (note: Validately can help with that)
- Run the tester through a prototype and ask them to complete specific tasks.
- Ask Real Life Usability questions noted above.
- Take notes and record the session to share important parts with stakeholders
- How many testers: Try to speak with 5 testers per persona. If you don’t have time or budget for 5, then 3 is better than zero.
Unmoderated Testing is asynchronous. You set up a test, send it out to people and record their interactions and often voice as they attempt to complete certain tasks. You are not live with them.
- Benefit:
- It is quick because it is asynchronous. You can get large volumes of responses while you are working on something else.
- Tradeoff:
- Can’t ask follow up questions and dig in while a user is in the act. Thus you can really only learn UI Usability. Unmoderated is NOT helpful for learning Real Life Usability.
- How to set up this test:
- This test is best done with a clickable prototype, because you are certainly going to learn and want to make changes.
- Recruit testers that meet your persona from either a panel or on your own.
- You give test takers a task to complete on the clickable prototype and measure how many can complete the task within a reasonable time period. Often you want the participant to share their thoughts out loud as they attempt to complete a task. You can ask qualitative questions afterward such as the SUS scoring approach.
- Tasks should not be overly prescriptive and should not use specific words that are used in the UI. Example:
- Bad task:
- Go to the Dashboard and “Add A New Story” to the queue. [since there are buttons labeled Dashboard and Add A New Story in the UI]
- Good task:
- Add a story to your backlog
- How many testers: There are two approaches with unmoderated testing:
- High volume click tests:
- typically 100 respondents and you just look at click patterns and qualitative responses
- Talk-alouds:
- this is when the users voice their thoughts outloud. You only need 5 testers per persona to learn 85% of the usability issues.
SJ: What do product leaders need to know about the kind of testing Validately offers?
SC: When we speak with Product Managers, UX Designers and User Researchers about why they don’t conduct more User Research/Testing, they usually say the same thing... I don’t have the time! When we dig into this, it really is because the process around the actual test sessions take longer than the testing sessions. Specifically, there is great pain around:
- Finding representative customers (Personas) to take tests
- Scheduling these tests
- Running these tests without fooling around with recording software
- Producing post-test reports in a shareable manner for stakeholders
The best practice is to have all of the key stakeholders involved in user testing sessions. Usually a User Researcher leads them, but Product Managers, UX Designers and Developers should sit in on the tests. While this might sound like a big time commitment, it actually saves time because it removes the needless debate on feature designs. The customer’s word is heard by everyone.
If you do not have a User Researcher on the team, then the session is usually led by the UX Designer. The PM should sit in on the sessions to learn as well, but you want to be careful to not have a bunch of moderators peppering the tester. One Moderator only.
About Steve Johnson
Steve Johnson is a product management process coach, author, speaker, and advisor. His approach is based on the belief that minimal process and simple templates result in a nimble product marketing and management team. Before founding Under10, Steve was a Pragmatic Marketing instructor for over 15 years and personally trained thousands of product managers and hundreds of company senior executive teams. Learn more at www.under10playbook.com
In his spare time, Steve enjoys making music, and has an album available on iTunes. Check out Steve’s music at notExactlySteve.com