1. Data Sources are Proliferating

Organizations are analyzing data from a wider variety of sources than ever. This is fueled by the increasing use of SaaS applications across business functions and accelerated by the ease of moving data from these apps into data warehouses or data lakes using SaaS ETL tools. This new paradigm allows companies to use best-of-breed tools for demand generation, customer marketing, salesforce enablement and customer support, and still be able to assemble all of the touchpoints for a given customer into a unified picture of their lifetime customer journey.

2. Machine Learning has Matured

Machine learning, from algorithms to engineering practices, has matured in a way that allows data cataloging tools to include ML capabilities that add useful context to, and recommendations within, an analyst’s workflow. Previous generations of tools often felt like completely different systems, needed specialized knowledge to use, or provided little more than quickly out-of-date documentation. As a result, data catalogues have often failed to gain widespread usage by the people who could benefit from them most. The newer generation of tools promises to meet the user where they are already working, for example in the SQL query interface, and be as easy (and informative) as Slacking a coworker.  

3. Engineering and Data Talent are in Demand

Demand for talent and the widely covered impact of the “great recession” reinforces the importance of data catalogs in facilitating knowledge transfer. Data employees tend to develop deep tribal knowledge, often in opaque but critical pieces of the data foundation. Nuances related to when an upstream process changed, the correct filters to apply to obtain financial numbers, bespoke code that sits behind management dashboards or production models: there are many reasons these employees (and their knowledge) are valued. In our experience, acquiring this knowledge is often a slow process for new employees. Knowledge transfer can be tedious for tenured folks. And with the seemingly insatiable demand for new data and engineering talent and the well-documented volume of employee turnover across industries, your team may run the risk of losing unknown amounts of specialized knowledge for good. Data catalogs can help solve some of this pain by storing in code and presenting in convenient interfaces what has previously been locked away in the minds of a few. They promise to accelerate time to impact by helping new team members find the data they need faster and more accurately and reduce the some of the frustrations of working with data that can themselves contribute to employee turnover.