Modern companies often deal with large and complex data sets from different and possibly unrelated data sources (CRM, IoT, streaming data, marketing automation, finance, etc.). Large companies often have branches in different geographic locations. This can complicate the process of data using or storing (in the cloud, hybrid multicloud, on-premises, etc.). Data Fabric will help to combine data from different sources and repositories, transform and process it for further work. As a result, users get a holistic picture of the current situation, that allows them to explore and analyze data to conduct effective business activities.
Data Fabric is a data integration architecture using metadata assets to unify, integrate, and manage disparate data environments. The main task of Data Fabric is to structure the data environment, and it doesn’t require replacement of existing infrastructure. Metadata and data access are managed by adding an additional technology layer over the existing infrastructure. Standardizing, connecting, and automating of Data Fabric data management practices improves data security and availability, enables end-to-end integration of data pipelines and on-premises cloud, hybrid multicloud, and edge device platforms.
Benefits of Data Fabric using:
- consistency across integrated environments through metadata management, knowledge graphs and machine learning;
- a holistic view of the current business situation, allowing users, analysts and data scientists to find relationships;
- maximize hybrid cloud capabilities and reduce development and management time for integration, deployment, and maintenance by simplifying infrastructure configuration;
- a simple process of studying and analyzing data for users without the participation of IT-specialists;
- availability of all approaches to data delivery through support for ETL, data virtualization, change data capture, streaming and APIs;
- automating routine tasks (for example, schema matching with new data sources and dataset profiling).
Data Fabric simplifies a distributed data environment where they it can be received, transformed, managed, stored. It also defines access for multiple repositories and use cases (BI tools, operational applications. This is made possible by continuous metadata analytics to build the web layer. It integrates data processing processes and many sources, types, and locations of data.
Differences Data Fabric from the standard data integration ecosystem:
- Enhanced data catalog that includes and analyzes all types of metadata;
- Knowledge graphs showing relationships between data elements (concepts, objects, events, etc.);
- Metadata activation – using machine learning active metadata management allows to create and process metadata on a large scale;
- Data preparation and ingestion – support for all approaches to data preparation and delivery including ETL, ELT, data streaming, application integration, data virtualization;
- Recommendation engine – constant analysis, study and formation of recommendations and forecasts in relation to integration and data management;
- DataOps – bringing the DevOps team together with engineers and data scientists to meet needs.
The Data Fabric architecture depends on individual data needs and queries of business. However, there are 6 main levels:
- Data management (ensuring management and security processes);
- Receiving data (determining the relationship between structured and unstructured data);
- Data processing (only relevant data extraction);
- Data orchestration (data cleansing, transformation and integration);
- Data discovery (identifying new ways to integrate different data sources);
- Access to data (the ability of users to explore data using BI tools).
When implementing Data Fabric, you need to consider:
- collection and analysis of all metadata types;
- transformation of passive metadata into active ones;
- creation and management of knowledge graphs;
- providing a solid basis for data integration.
DataLabs is a Qlik Certified Partner. A high level of team competence and an individual approach allows to find a solution in any situation. You can get additional information by filling out the form at the link
