Data mesh is an approach to creating a decentralized data architecture. It is based on domain-driven design. In data mesh, development teams are given responsibility for the development and operation of data products of their business domain. Data Mesh transfers the responsibility for data analysis to the technically cut domain teams. Analytical data is also made available to other teams in the form of data products in order to enable comprehensive evaluations. Common standards and rules are jointly defined by the teams in the form of federated governance to ensure interoperability and security requirements. The data infrastructure is provided in the form of a self-service data platform by a data platform team.
In the data mesh concept, specialist departments (like marketing, sales, and customer service) produce and consume data from other parts of the company. To ensure seamless transfer of information, employees should comprehend what good data handling entails and its impact on the organization. Training programs that provide instructions on standards, use and best practices can be instrumental in developing a fundamental understanding and culture pertaining to handling data. Moreover, specific training courses covering topics such as data sharing, quality and governance are beneficial too. There are certain principles that must be followed when implementing a data mesh structure. Data Mesh is based on the following four fundamental principles:
- Domain Ownership
- Data as a Product
- Self-Serve Data Platform
- Federated Governance
A transformation should prioritize areas of higher maturity that do not have a major impact on IT. An agreement between departments is crucial to steering the early phase of the transformation as the demand for data specialists increases.
---
It’s important to consider that data mesh encourages the adoption of cloud native and cloud platform technologies as a means of scaling and achieving data management objectives. This concept is often likened to microservices, helping listeners appreciate its application in this milieu. Seeing as the distributed architecture is apt for scaling data requirements throughout a business, it follows that a data mesh may not be suitable for all types of companies. In particular, smaller businesses may not benefit from a data mesh due to their less complex enterprise data than bigger firms.
While domain teams play a bigger role in managing ETL pipelining under this model, this does not minimize the role of a central data engineering team – instead, it now focuses on finding the optimal infrastructure solutions for their products stored there.
In a way analogous to how microservices enable applications to offer business or consumer-oriented functions, a data mesh uses functional domains to set up parameters for the data, treating it as a product with access rights throughout the organization. This increases data integration flexibility, permitting users to extract and make use of data from multiple domains simultaneously for tasks such as analytics.
