Turning Product Usage Data Into User Features

Problem

Gaining a precise understanding of how users interact with product features is key for product managers to build products, users will engage with and love to use. But doing so at scale and taking action on what can be learned from analyzing product usage data whereas in B2B or B2C environments can be extremely challenging.

As the Sr Director of the Product Analytics for Autodesk, my team of product analysts helped me realize that many product managers across product lines were struggling to get a holistic understanding of product usage data despite having on-demand access to well-structured user-level data describing the sequence of commands and operations end-users are going through.

Actions taken

We soon realized that the best way to tackle this analytics gap was to bring a user-centric perspective to it and handle this project as a customer-facing service. We then assembled a small product growth-led team and kicked-off a project focused on turning product usage data into valuable and personalized insights for end-users, aimed at improving user proficiency with our products.

Over the course of one year, we created an insight platform and business processes to ingest product usage data, run signal detection algorithms and ML models to identify out-of-norm usage patterns and deliver insights matching those situations. The system is now in production and processes billions of user transactions weekly to ultimately deliver millions of personalized insights to end-users.

Lessons learned

• Delivering insights that are truly insightful requires to put the end-user front and center. They are the ultimate judge of how insightful your advice is and as such: time, rapid iterations and the ability to process closed-loop data are key criteria for success. Trust can only be established over time and insightfulness is not a single event.
• Product usage data is both very powerful and very sensitive. Even in the context of the very complex CAD products of Autodesk, our performance metrics indicate that the vast majority of our users find our insights to be relevant to their product usage and accurate based on their proficiency levels. However, this alone won’t be enough, and you will also have to get your message delivery right because being right on the insights but wrong on how you deliver them would be such a waste of user attention.
• Putting together a platform for delivering insights at scale is not for the faint of heart. It takes quite a few technology wizards to get things right and a solid technology infrastructure. But quality instrumentation is the prerequisite to the whole journey, or you will ultimately prove on your boss’s budget that the garbage in – garbage out adage is indeed true.