Inspiration: I talked to a guy...

“I talked to a guy” isn’t a statistically valid sample.— Steve Johnson

In a workshop, we were discussing customer interviews and someone wondered why we bother. “After all,” he said, “people only seem to listen to data. They want to know how many and what percentage.” He suggested that we stop doing interviews and focus our energies on surveys instead.

Except… you don’t know what you don’t know.

For years, I ran annual surveys to understand how product managers spend their time, how they’re compensated, and what issues they’re facing. But I was able to know what questions to ask because I had lots of customer experience and interviews.

I had a series of hypotheses that I needed to support with data.

It’s helpful to understand the differences between qualitative and quantitative research. With qualitative research (like interviews), the focus in on open-ended questions—questions that may go down paths that you didn’t necessarily expect. This type of research takes awhile; you may spend 30 to 45 minutes; you may spend a half-day. You’ll need a skilled interviewer (like a product manager) to do these successfully. Interviews and discussions give you insights that you and your colleagues wouldn’t have encountered in your office.

Quantitative research (like surveys) is about the numbers: what percentage has this problem? How many struggle with this issue? These surveys can be conducted by a telemarketing firm or via a web survey because they don’t require a skilled interviewer.

One of my favorite Innovation Games is “Me and My Shadow.” It’s a method for observation to understand a customer’s workflow and interactions with your product. You’ll see a dozen things that you want to address—new features as well as design changes.

You’ve probably done this in another way: helping your parents. It’s amazing what you see when you watch someone else use a program. You ask, “Why did you do that?” a lot. And in many cases, you’ll find yourself wondering, “Why did the developer design it that way?”

Observation is a form of qualitative research. So are discussions and interviews.

But “I talked to a guy” isn’t a statistically valid sample. It’s the first step in forming a theory. Now, make it statistically relevant by backing it up with a survey.