User Testing to Minimize Time Spent on Arguing Conceptual Ideas

Problem

The problem was that multiple people across the organization felt that they knew what would be better for our customers on our key landing pages, but they didn't agree or have data to support. Most of the suggestions were based on theoretical assumptions of what they liked or had heard from a small sample of customers. Being a small company, we couldn't rely on quick A/B testing to help us end these debates, like Facebook can for example, so we needed to find a better way to make UX decisions.

Actions taken

"The answer was to ask customers."

It sounds obvious that the answer was to ask customers, but in our business we try not to bother customers who have often just recently lost a loved one. So, the next best option was to get a subscription to UserTesting and run a series of studies to put the site in front of perspective customers and get their feedback directly.

In user testing, you usually do five people at a time because you're looking for errors, not sentiment. So, in order to get a broader set of users, I used the same test with a variety of user segments. I would take five men, five women, five on mobile, and five on desktop, and then ask them all to respond to, and rank the UX, on a particular measure like "trust" or "affordability", whatever the question was we were trying to solve for.

From there, I was able to see enough trends in reactions, and how those reactions affected the way they ranked the UX on those attributes. From there, I could inform our stakeholders from an informed perspective. It doesn't provide the quantitative value of an A/B test, but does yield valuable qualitative data to inform the decision.

Lessons learned

• Often, arguments about conceptual ideas don't bring people together. In fact, it can cause people to dig in their heels on their perspective. In the end, it doesn't matter who was right & who was wrong; what matters is how well we're able to predict how customers will respond. These arguments still happen, but now we have a way to keep the emotion out and shift the discussion to "what do we want to know".
• Implementing user research takes both time and money, but it takes less of both than making bad decisions and only learning through implementation. I can launch a study on user testing at 5 p.m. on a Tuesday and by Wednesday at noon have usable data.
• Publishing the study results helps to ground everyone in the experience of real users, rather than hoped-for results. It helps everyone to hear the words of real people who are interacting with our products and understand that those reactions are both varied and unexpected.