"Data catalogs undergo a shift in perspective at Forrester; here are the essentials"
The data industry is undergoing a significant transformation, moving away from siloed, manual, and fragmented metadata management towards unified, automated, and AI-augmented metadata ecosystems integrated across diverse data platforms and environments. This shift is aimed at creating a single control plane for metadata, consolidating definitions, data lineage, classifications, and access policies to enable real-time governance, compliance, and operational agility.
Gartner has scrapped their old categorization of data catalogs in favor of a new one that reflects this shift. The solution to the growing needs of metadata management is "active metadata", a key component of enterprise data catalogs for DataOps. These catalogs, such as the emerging Enterprise Data Catalogs (EDCs), handle the diversity and granularity of modern data and metadata, acting as a "system of record" to automatically capture and manage all of a company's data.
Forrester Now Tech: Enterprise Data Catalogs for DataOps, Q1 2022, states that provisioning data is more complex under distributed cloud, edge compute, intelligent applications, automation, and self-service analytics use cases. To address these challenges, EDCs provide granular data visibility and governance with features like column-level lineage, impact analysis, root cause analysis, and data policy compliance.
Active metadata makes a two-way movement of metadata possible, sending enriched metadata and unified context back into every tool in the data stack. This enables powerful programmatic use cases through automation, such as automated flags, alerts, and suggestions to help users keep on top of complex, fast-moving data flows.
EDCs are designed around modern DataOps and engineering best practices, connecting the "data and developer environments" with features like bidirectional communication, collaboration, and two-way workflows. They enable delivery across data engineers, data scientists, and application developers, creating data transparency and enabling data engineers to implement DataOps activities.
The new concept of metadata is not the old-school version that is slow and siloed. Instead, it is a dynamic, active, and AI-powered approach that reflects the needs of the modern data landscape. The data industry is experiencing a shift in how metadata is perceived and managed, with new ideas such as the metrics layer, modern data catalogs, and active metadata emerging in the past year or two.
The focus on metadata has increased significantly in 2022. The data industry is currently shifting from siloed, manual, and fragmented metadata management towards unified, automated, and AI-augmented metadata ecosystems integrated across diverse data platforms and environments. This new approach centers on creating a single control plane for metadata that consolidates definitions, data lineage, classifications, and access policies across cloud, on-premises, and SaaS sources to enable real-time governance, compliance, and operational agility.
For those interested in learning more about active metadata, DataOps, data culture, and the author's experiences building Atlan, they can subscribe to the author's newsletter, Metadata Weekly. The author writes weekly on these topics.
The shift towards active metadata marks a significant change in the metadata space, moving away from old-school data catalogs and embracing the future of metadata. Businesses are investing heavily in this transformation, with the metadata management tools market projected to grow from $10B in 2025 to $30B by 2033, driven by these innovative concepts and tools.
Sources:
- The Forrester Wave™: Enterprise Data Catalogs For DataOps, Q1 2022
- The Rise of Active Metadata
- The Metrics Layer: The Next Evolution of Data Catalogs
- Federated and Privacy-Preserving Governance in Modern Data Catalogs
- Data Fabrics and Unified Data Ecosystems: The Future of Data Management
- The Data Observability Integration: A New Era for Data Management
- The new approach in metadata management is moving from the traditional, siloed methods towards unified, automated, and AI-augmented data-and-cloud-computing technologies, as reflected in the shift towards active metadata and the rise of modern data catalogs.
- For those seeking to advance their education-and-self-development in data-and-cloud-computing, learning more about active metadata, DataOps, data culture, and innovative concepts like the metrics layer can be achieved by subscribing to resources such as the author's newsletter, Metadata Weekly.
