What is data fabric?

How does data fabric affect your data estate?

A data fabric is an information architecture that unifies data across an organization. According to Techopedia, data fabric describes “a distributed IT architecture in which data is governed the same way whether it is located on premises, in the cloud, or at the edge of a network”.¹

Rather than being a single piece of technology, data fabric is a holistic data and artificial intelligence (AI) strategy that helps organizations leverage all existing and future investments within their data estate. The purpose of a unified data fabric is to ensure that an organization's data is always accessible to all authorized parties, regardless of where it is stored.

In the Gartner article Data Fabric Architecture is Key to Modernizing Data Management and Integration, author Ashutosh Gupta notes that data fabric “leverages both human and machine capabilities to access data in place or support its consolidation where appropriate [and it] continuously identifies and connects data from disparate applications to discover unique, business-relevant relationships between the available data points”.²

Data Fabric for Dummies author Ed Tittel states that “data fabric is a living, always-evolving collection of capabilities that grows and changes along with whatever organization it serves”.³

Why is data fabric important?

One of today’s biggest business challenges is figuring out how to close the critical gap between the available data and information, and transforming as much of that data and info into resources like knowledge and insights. These insights are necessary for creating personalized, compelling customer experiences, and cutting edge products and services that enhance a business’ operational efficiency.

As technology becomes more scalable and people are more connected, the complexity and gaps in leveraging actionable insights increase. According to Forrester, “an average of between 60% and 73% of all data within an enterprise goes unused for analytics”.⁴ Today’s businesses struggle to close that gap.

Data Fabric as Modern Data Architecture author Alisa LaPlante notes that the explosion in the growth of data and continued fragmentation has almost rendered data centralization a distant dream. LaPlante points out that “with data being generated and stored everywhere from the data center to the edge to the cloud, having a single centralized [system of] record doesn’t work anymore”.⁵

According to LaPlante, “data fabric helps businesses virtualize their data, instead of relocating it to a centralised location. Security controls and authentication measures can be applied as though they were all in one place. It also makes things much easier for the company's users. Virtualizing, not centralising, is the foundation for a strong data fabric”.⁵

LaPlante argues that data fabric offers the following benefits:

• Data for all scenarios and users by providing timely, reliable, and reusable data for a variety of analytic, operational, and governance use cases, along with engaging business self-service users.
• Data from across sources by leveraging metadata, models, and pipelines, accesses, merges, and transforms both in-motion and at-rest data from across a heterogeneous, dispersed data landscape.
• Data that spans any environment and allows for a flexible spanning of data if it's the distributed via an on-site environment, hybrid environment, or multicloud environment.   

Data fabric is a gateway for businesses looking to move beyond older data management practices and optimize their resources. According to a StrategyR market report, “the market for software products and services that facilitate the creation and management of data fabrics will grow to be $3.7 billion annually by 2026”.⁵

What are the benefits?

Data fabric is often viewed as a gradual offspring of earlier legacy systems with hardware and a network but no overarching network management system for getting data where it needed to go.

Forrester finds that a data fabric “minimizes the complexity [of a data estate] by automating processes, workflows, and pipelines, generating code and streamlining data to accelerate various use cases”.⁶

In Data Fabric for Dummies, author Ed Tittel declares that “data fabric brings worthwhile change and numerous benefits to those who buy into the vision”.³ Here are several examples of data fabric's benefits:

• Accommodates different environments – Ed Tittel states “a data fabric can accommodate multiple instances of all kinds of execution environments, including on-premises data centers, cloud platforms, and edge systems”.³
• Process data regardless of velocity – According to Tittel, “a data fabric can process and provision data at all velocities from streaming data in real-time to scheduled batch jobs (regularly or infrequently, large and small, fast and slow)”.³
• Reduces time to insights – From the bmc blogs post “Data Fabric Explained: Concepts, Capabilities & Value Props”, author Muhammad Raza states that data fabric helps to “mitigate disruptions from switching between cloud vendors and compute resources to process data stored in disparate locations, [which] reduces time to insights dramatically”.⁷
• Governance and compliance – Raza notes that data fabric helps unify the overall security and governance process by ensuring that it’s “centralized and consistent across all environments and making it possible for “local management of metadata in compliance with global organizational policies [to] be applied to all data assets”.⁷
• Data transportation and processing made easy – Tittel notes that data fabric “can move data from platform to platform for ready consumption or storage and archiving [with] no extensive refactoring needed”.³ When it comes to moving processing, data fabric can “[shift] processing from one execution environment to another without extensive recoding”.³  

Hybrid Cloud & Data Fabric for Dummies authors Larry Freeman and Lawrence C. Miller suggests that data fabric is an infrastructure “built for today, but designed for tomorrow”.⁹ According to Freeman and Miller, “as cloud popularity grows, [it is likely that] more and more corporate data will move into a shared cloud environment, with only the most sensitive data staying within the confines of the data center. A data fabric is designed to enable [and accelerate] this shift to the cloud”.⁹

What are the challenges for building a data fabric?

While the benefits of data fabric are multifold, there are many blockers that prevent organizations from successfully adopting a solid data fabric architecture. 

According to LaPlante, “many companies spend a lot of time, effort, and money [attempting] to figure out if they can access their data, as well as define and add value to it. However, they don't always use analytics to acquire insights. And almost all of them fail to apply what they've learned and [then optimize to] shut the loop. When data fabrics become commodities, how well organizations execute in a closed-loop analytics system will determine how they compete in the future”.⁵

Challenges to data fabric adoption include, but aren’t limited to, the following:

Modernization
Organizations today often own and use their data assets. A data fabric helps to unify all these assets, creating a “consolidated data management environment that extends across an organization’s edge-to-core-to-cloud infrastructure for all platforms and applications”.³

Tittel argues that data fabric implementation usually requires a modernization process of “[taking] existing fragmented and siloed approaches to managing, storing and situating data into the data fabric’s single, consistent, policy-driven, self-service environment, supported by DevOps and DataOps principles”.³

Tittel states that “modernization is a vital job for organizations seeking to deploy a data fabric [and that it] involves nothing less than a review of all data assets relevant to a business case, and all applications and services that consume and produce data [while trying to open] the data fabric’s umbrella to cover everything”.³

The human component 
Data fabric, like so many other technology-based elements in business and the world today, has a human factor. Tittel notes that “applications come with stakeholders, users, and developers, all of whom must understand and buy into that fabric [and enterprises] risk fighting “shadow IT” and other end-arounds that favor “quick and dirty” over consistent and coherent”.³

Investment
Tittel notes that following the financial crises and the pandemic, there’s increased “demand that any technology investments provide a faster [ROI which] puts added pressure on existing investments [and] it encourages cautious and critical consideration of new [investments]”.³ He argues that the market conditions “[benefit] use cases at the high end of the volume and variety curve where bigger payoffs prevail”.³

Agile methodology
Tittel notes that for data fabric adoption, modern data fabrics requires the enterprise to leverage DataOps and must use the “CI/CD [continuous integration, continuous delivery] Agile methodology [that] requires data analysts, business stakeholders, and IT personnel to stop working in disparate environments [and] collaborate to ensure [data comprehension and] requires data analysts, business stakeholders, and IT personnel to stop working in disparate environments”.³

Tittel notes that this integration will produce “self-monitoring and self-measurement for the data fabric so that it keeps getting better at managing data through its entire lifecycle”.³ By this logic, the data fabric is in a persistent state of optimization, and is constantly performing better than it had been previously.

How to optimize your data fabric and help it generate value

Data fabric helps you make better decisions through improved compute performance across data channels. Ashutosh Gupta argues that “to deliver business value through data fabric design, D&A leaders should ensure a solid technology base, identify the required core capabilities, and evaluate the existing data management tools”.²

Data fabric collects and analyzes metadata

Gupta notes that “Contextual information lays the foundation of a dynamic data fabric design. There should be a mechanism (like a well-connected pool of metadata) that enables data fabric to identify, connect, and analyze all kinds of metadata such as technical, business, operational, and social”.²

Passive metadata must be converted into active metadata

Gupta argues that for data sharing to work best, enterprises must activate their metadata. For this metadata to become active, their data fabric must do the following:

1. Locate and analyze the key metrics’ and statistics’ metadata
2. Construct a graph model
3. Graphically depict the metadata as it relates to the enterprise’s relationships
4. Use decisive metadata metrics to enable AI and machine learning (ML) algorithms
   • These AI/ML algorithms will gradually learn and generate predications for data management, integration, and more²

Knowledge graphs help enrich data with semantics

Gupta notes that the reason data fabric needs to make and curate knowledge graphs, is because these graphs “[enrich] data with semantics [and help] data and analytics leaders to derive business value”.²  He states that the “semantic layer of the knowledge graph makes it more [intuitive and] the analysis easy for D&A leaders”². Lastly, the semantic layer “adds depth and meaning to the data usage and content graph, allowing AI/ML algorithms to use the information for analytics and other operational use cases”.²

Data fabric requires a robust foundation of data integration

One of the benefits of data fabric adoption is how flexible it is in terms of data delivery. It easily works with multiple data delivery methods, which help its enterprise to accommodate a vast array of data consumers, extending from traditional IT through finance and business.