Managing Data Investigations: Tips for Engineering Managers

Introduction

In this article, we will discuss strategies for effectively managing data investigations, specifically focusing on tips for engineering managers. We will explore how to stay focused on the core issue and avoid scope creep. To provide valuable insights, we will draw from a conversation between two top-tier engineers, Indrani G and Sarah Phippen.

The Importance of Asking Insightful Questions

Sarah, a newly promoted manager in a data engineering practice, seeks guidance on how to conduct interviews for hiring a data analyst. She wants to ask insightful questions that go beyond the typical resume walkthrough. Indrani, an experienced engineer, offers her advice:

"I would recommend asking candidates about instances where they investigated something and brought insights to the table. This question allows you to gauge their problem-solving skills and their ability to think critically."

Staying Focused on the Core Issue

Sarah explains that her team often faces data quality issues, such as missing or duplicated customer information. She wants to hire someone who can investigate these issues and stay focused on the core problem. Indrani suggests:

"Being able to prioritize tasks and stay focused on the committed problem is crucial. You want someone who can resist the temptation to expand the scope of their investigation too quickly. It's important to fix one issue at a time and then move on to the next."

Effective Stakeholder Management

Sarah mentions that her team works with various stakeholders, including the data science team, operations team, and business teams. Indrani emphasizes the importance of effective communication with different stakeholders:

"Being able to communicate effectively with stakeholders at different levels is essential. The information needed by the data science team may differ from what the operations team requires. Look for candidates who have experience working with different stakeholders and adapting their communication style accordingly."

The Value of Tooling and Automation

Indrani suggests considering candidates with experience in catalog tools, customer data platforms (CDPs), and data governance tools. These tools can help with data reconciliation, data quality, and data lineage. She also recommends asking candidates about their experience with automation and libraries:

"Look for candidates who have implemented automation or used libraries for data quality checks. Python libraries like Great Expectations and DQ can be valuable in ensuring data quality. Additionally, candidates with experience in CD tools or data governance tools can bring valuable insights to your team."

Migration Experience as a Signal

Indrani highlights the importance of migration experience when evaluating candidates:

"Consider candidates who have been part of migration projects. These projects require problem-solving skills, data reconciliation, and collaboration with different teams. Candidates with migration experience often have a broader understanding of data and can bring valuable insights to your team."

Conclusion

Managing data investigations requires a combination of technical skills, problem-solving abilities, and effective stakeholder management. By asking insightful questions, staying focused on the core issue, leveraging tooling and automation, and considering migration experience, engineering managers can build a strong team of data analysts. Remember to prioritize growth opportunities for your team members and delegate tasks to foster their development.