Testing Product Ideas Using the Wizard of Oz Technique

Problem

"I led product for a company where we were making it easier to start a small business by taking the complexity out of Workers' Comp insurance. One of the major experience issues is that when a business buys a policy their price is based on an estimate of annual payroll. Of course, things change over the years and some businesses grow significantly. A traditional insurance company remediates this by performing an 'annual audit' and then billing the business owner for any payroll that was added over the year. The adjustment after the audit is often not communicated by the insurance company and comes as a complete surprise. Obviously no one likes surprise bills, least of all small businesses owners that have limited cash flow."

Actions taken

"We wanted to implement the simplest method possible. We thought of sending the customers an email with their current payroll and asking if it was still accurate. If not, we could run them through a workflow to collect updated payroll and adjust the remainder of their payments accordingly. More payroll would mean a higher price while less payroll would mean a lower price."

"However, there were a lot of questionable assumptions baked into this idea:"

• "Would customers respond to the email? They are busy small business owners, afterall."
• "Would customers understand why their price would increase or decrease?"
• "Our designer thought payroll was a sensitive number and that customers would be upset if we put it into an email. We couldn't say that she was wrong."

"I determined that this was a good time to employ a product discovery technique called the Wizard of Oz. The concept is that you use a facade on the front-end with humans behind the curtain to manually perform any backend processes."

"So we ran the following test:"

• "We emailed 300 customers who had been with us for at least 3 months using a free Mailchimp account."
• "The email showed their current payroll and contained two CTAs (call to actions) - 'My Payroll has changed' or 'Nothing's Changed.'"
• "Customers who clicked 'Nothing's Changed' were simply directed to a Thank You page."
• "Customers who clicked 'My Payroll has changed' were sent to a Typeform (a fancy survey tool we used as a stand-in for a UI) link and were asked to provide updated payroll and number of employees."
• "In addition, we paired this with 5x moderated usability tests to:"
  • "see if customers could complete the workflow, and"
  • "To assess if they understood why their prices were increasing or decreasing."

"The results:"

• "We achieved a 70% response rate."
• "Several customers emailed us to say that they really liked this feature because it made them feel like 'we were watching out for their business.'"
• "Usability studies confirmed customers understood that adding to or decreasing their payroll would affect their price accordingly."

Lessons learned

• "Problem solving is about articulating a set of assumptions and running tests to validate/invalidate each one."
• "The slowest way to test assumptions is to create scalable, shippable code. Instead, use a combination of the many product discovery techniques to test assumptions as quickly and cheaply as possible."