One Approach for Two Data Platforms
Supporting Architectures for Multiple Data Platforms
For different reasons enterprises support two or even more central data platforms. Some typical reasons are:
Merger from two companies
Companies with rather decentral organisations establish a second platform
Platforms are initial for specific reasons but grow to a all purpose data platform over time
Companies start migrations from one to another but never end it
etc.
How it started…
My customer come from a rather decentralized approach and is on the way to establish Data Mesh as a socio-technical data architecture.
The two main platforms SAP BW/4HANA and Snowflake need same data and different approaches like direct data exchange, establishing an enterprise bus with Apache Kafka and others are discussed. On possible approach is to solve it via architectural thinking.
How it is going…
While Data Mesh already support decentralized architectures it can easily be combined with approaches like Data Fabric and Data Hub. While in Data Fabric data can be combined and provided from different data platforms, it sometimes comes to limitations through virtualization and when it comes to complex calculations or historizations of combines datasets. A Data Hub is a data distribution approach where Data Mesh can be used as a form of Data Hub or the Mesh is build upon a common data layer provided by the Data Hub in a multi-platform Data Mesh-approach.
Outlook
Solving complex data landscape issues with architecture can lead to a more efficient architecture and better data democratization. Direct exchange of data is always possible but leads to complex maintainance situations over time and often needs 3rd party solutions to handle bi-directional data exchange.