How a sandwich shop explains the value of observability in IT operations

By Tiago Dias Generoso

My favorite sandwich shop smells like sizzling bacon and fresh-baked bread.

The small shop hums with activity, seven days a week. People go for the classic burger that’s topped with cheese, bacon, and a fried egg.

The sandwiches are delicious, but what most impresses me is how the shop maintains such a seamless operation. Happy customers benefit from the orchestration of everything from inventory to ordering to assembly and checkout to quality control and more. It makes me think about observability in IT operations for large companies.

No matter the industry, organizations need to understand customer behaviors, preferences, high-demand periods, bottlenecks in the system and other factors that impact the bottom line. Data inputs about these factors drive business outcomes, but I see many companies still struggle with how to get the most from their data. That’s why I’m so passionate about the concept and practice of observability in IT operations.

Observability enables your team to have a clear view of your company’s technology ecosystem. It empowers tech teams to spot issues, optimize performance, cut costs, reduce carbon footprints, and foster innovation, while helping to ensure a seamless customer experience.

In an environment where CTOs must deliver business results through technology, observability becomes IT secret sauce. (Sandwich pun intended.)

Observability is more than monitoring

One of the common misperceptions about observability is that it is merely monitoring. It isn’t. Observability is about collecting data from various aspects of your system, and harnessing machine learning and AI for actionable insights about it. Observability makes monitoring contextual.

Observability building blocks

The building blocks of observability are traces, metrics, and logs. Traces are particularly powerful. A trace shows how a request or task moves through the many connected parts of an IT system. Traces show the flow of data and processes, and how different components and services interact.

At the sandwich shop, a trace ID may be like a table number. Everything that occurs during your visit gets associated with your table number. Similarly for your IT operation, traces link metrics and logs of a user session. This can be crucial for identifying bottlenecks and improving user experience. The technical team can use traces to pinpoint and resolve problems more efficiently, reducing the time and resources needed to discover and address issues.

In the context of a web page, for example, a trace may enable the team to identify the application response time and gain insight into the experience of a customer who is using the web page. Without traces, and an observability practice, the downstream impact of poor system performance or errors may not be clear.

Five myths about observability in IT operations

Myth 1: Observability is always expensive
This is simply not true. Several cost-effective observability solutions, including many open-source options, are available. Costs vary depending on automation requirements, the intelligence behind it, ease of implementation and other factors. Think of observability as an investment that pays off by helping you avoid costly outages and performance problems.

Myth 2: Observability can be achieved by installing a tool
Yes, there are observability tools, but observability also requires new ways of working. Observability tools will not make a significant impact unless they are incorporated with operating models and company goals.

Myth 3: Observability is only for big enterprises
Outages, performance problems, and operating costs are significant issues for organizations of all types and sizes. Observability can help regardless of industry.

Myth 4: Observability is only for technical teams
By understanding how your systems work and their performance, everyone from managers to sales teams to customer service reps can make better decisions.

Myth 5: Observability and data observability are the same
While the core idea of making things “observable” is the same, data observability primarily focuses on ensuring the quality of data generated by IT. This quality data enables organizations to make informed decisions. Data observability aims to pinpoint root causes behind the absence of high-quality data, which may include human errors, data corruption, system malfunctions, or other factors.

An observability success story

As an example of observability in action, consider the experience of one customer which faced the complex task of managing 1,600 business applications for over 280,000 global users. To help meet the challenge, the company moved to a hybrid cloud ecosystem.

Given the labyrinth of components, pods and containers, the company needed to make observability part of the program. Otherwise, it would have been impossible to understand performance across the hybrid cloud infrastructure, streamline operations, or optimize resources.

A symphony of observability, AIOps, sustainable IT, and FinOps practices has driven dramatic results for the customer.

• Speedy resolution of application and infrastructure tickets
• Accelerated root cause analysis
• Reductions in resource consumption and carbon footprint

Call to Action: How to get started

Step 1. Assess your observability readiness by understanding the tools and technologies used by the company, how IT data is used, quality of the data, and whether the company is satisfied with the performance and reliability results. This will help you prioritize the focus areas for starting an observability program

Step 2. Define your observability goals. What do you want to achieve with observability?  For example, use IT data to make better decisions, reduce costs, improve the performance and reliability of the systems, improve customer satisfaction, and align business and technology.

Step 3. Choose the right observability tools and technologies. Your needs, budget, team skills, and experience with the technology will all factor. Do your vendor research, ask for success cases, try the free trials, and create tooling comparisons.

Step 4. Anticipate bumps in the road. Observability is not a one-time project and requires a culture change. You’ll need executive and team buy-in to adopt new ways of working. That might not happen out of the gate.

Observability is a long game. Embrace it to help your organization stay resilient and customer-focused, and just like a well-run sandwich shop, you keep customers returning for more.

Tiago Dias Generoso is a Distinguished IT Architect | Senior SRE | Master Inventor based in Pocos de Caldas, Brazil.