Why (and how) to make a business case for data observability

By: Rajeev Puri, Naveen Kamat, Karunakaran Samuel, and Shobhit Jaiswal

Picture yourself behind the wheel of a new sports car. If you want the vehicle to perform optimally, you’re not going to fill the tank with low-grade gasoline—you’re pumping in the highest-octane fuel you can buy.

The same concept applies to your company’s data. If you use the best information available to make decisions, you can expect premium results.

Unfortunately, many companies run their businesses with low-quality data and experience less-than-optimal performance because of it. In fact, bad data is one of the biggest barriers for organizations striving to become data-driven.¹

The operational impact of data deficiencies is the reason your technical teams should be talking about and looking to implement data observability into your data management strategy.

Changing how you view observability

Observability as you probably know it uses traces, logs, and metrics to monitor and manage your IT systems and applications. Data observability, on the other hand, describes your ability to understand the overall health of your data and data systems.

By collecting and correlating events across your data estate, data observability helps ensure your data is complete, accurate, and up to date.²  Simply put, better data helps yield better business outcomes.

We’ve distilled the advantages of data observability into three benefits to recast how you can look at, think about, and discuss this holistic approach to data management:

Data observability creates speed

Data observability’s value proposition is grounded in speed. Accurate information available in real time or as close to it as possible allows analysts and engineers to quickly discover and understand the impact any issues with the company’s data or data pipelines can have on their systems and processes. Faster discovery speeds resolution, which in turn saves time and money.

Imagine you’re a global manufacturer, and a data pipeline in your system fails, preventing you from sending and receiving data in real time. Without end-to-end oversight of the entire data pipeline and the ability to observe the overall health of your data system, it may take hours to detect and isolate data anomalies. So, what might otherwise be an easy fix could devolve into a major issue that disrupts the delivery and reliability of critical business reports that inform your production lines.

Meanwhile, data observability ensures you can quickly identify data irregularities and predict how that specific device going down will disrupt your data system. Then, based on the anticipated business impact, your customer experience officer (CXO) can decide whether to escalate the support ticket for immediate resolution or prioritize other duties.

In this scenario, full visibility into your data pipelines could theoretically reduce mean time to detect and mean time to restore by up to 60% and 80%, respectively. Faster anomaly detection and issue resolution should, by extension, improve your company’s ability to meet service-level agreements (SLAs).

Data observability creates simplicity

Data observability provides a continuous, unfettered view of your data ecosystem, making it easier to monitor, analyze, and act on data collected from across your organization. With complete transparency, engineers can track data as it flows through data pipelines and pinpoint issues that may affect quality.

Observability also contextualizes the data generated by your systems, providing insights into the “why” behind data aberrations. Engineers may then use these findings to resolve specific problems and design solutions to prevent similar errors from occurring in the future.  

By leveraging machine learning (ML) to simplify—and, in many cases, automate—workflow monitoring and anomaly detection, data teams can predict irregularities and respond to data incidents before they become serious issues. Less complexity should optimize pipeline performance, enhance security, and improve data reliability.

For example, one of India’s largest public-sector banks currently relies on a large data warehouse to store and process data needed for its regulatory reporting to the Reserve Bank of India (RBI). Issues with data quality and completeness have hampered reporting, leading to delays and error-prone reports being sent to the RBI.

To address these problems, the bank is implementing a unified data and artificial intelligence (AI) console that integrates data quality and governance features. The tool will facilitate data observability by enabling engineers to monitor all their data systems from a single pane of glass. This capability, in turn, will make it easier for data teams to identify quality issues at the data source, ultimately improving the accuracy of their reports and streamlining the reporting process.

Data observability creates savings

Data observability can profoundly impact your bottom line. By using the practice to improve data integrity, you facilitate:

• Predictive analytics. Coupled with machine learning, data observability allows data teams to detect issues in data sets faster and analyze their root cause more easily. Data observability also helps data teams discover and correct potential issues before they become problems.
• Targeted troubleshooting. When problems do occur, data observability provides the context data teams need to plan and prioritize remediation. This informed decision-making can reduce data downtime and associated costs.
• Data democratization. Making data more accessible to end-users throughout your organization—otherwise known as data democratization—helps boost operational efficiencies and improve regulatory compliance by unlocking valuable information stored in silos.
• Talent redistribution. Since data observability should reduce the time IT teams spend addressing data quality and availability issues, engineers can devote more energy to enhancing the overall performance and functionality of your IT systems.

In another customer example, one of Asia’s largest bioenergy producers was struggling to forecast yield, minimize plant stress, and monitor sugarcane crop progress. Unreliable data was a culprit. The challenges impacted product quality and increased production costs.

Recently, the company began using data quality and governance features built into a new data management platform to identify and address data quality issues. Greater visibility into the data systems helps engineers improve yield prediction and pinpoint areas in the field where production needs to increase.

The company also plans to offer an internal chargeback mechanism to business units based on their consumption of the data platform, enabling managers to monitor usage and reduce costs.

The bottom line on data observability

The quality of your data affects every aspect of your business. Shifting from traditional monitoring to data observability can help you maximize data quality and deliver positive business outcomes that keep your enterprise humming at peak performance.

Rajeev Puri is Vice President for U.S. Manufacturing, Communications and Energy Markets for Kyndryl and a Distinguished Engineer. Naveen Kamat is Vice President for Practice General Management. Karunakaran Samuel is Director of Data and AI Engineering and Services, and Shobhit Jaiswal is Director and Practice Specialist for Kyndryl Applications, Data, and AI.

¹ 6 barriers to becoming a data-driven company, CIO, May 2023
² Microsoft learn AI skills challenges, Microsoft, July 2023
³ On the general theory of control systems, University of Toronto archives, 2023
⁴ 5 pillars of data observability bolster data pipeline, TechTarget, January 2023