Unlock the power of generative AI for business with LLMOps and other keys

By: Naveen Kamat and Dennis Perpetua

A human wrote this article. A year ago, such a statement would have been unnecessary, even pedantic, but with the advent of AI, has it become necessary?

Expectations for using generative AI in business are bullish. Enterprise spending on generative AI solutions is forecasted to reach $143 billion by 2027.¹ McKinsey research predicts generative AI will add the equivalent of $2.6 trillion to $4.4 trillion to the global economy annually.² Companies eager to get started are left with a classic question: how is this going to generate real value for customers, employees and shareholders?

To unleash the full power of generative AI, we suggest businesses must address four pillars:

1. Build a strong data foundation
2. Leverage large language model operations (LLMOps)
3. Manage for potential shadow AI
4. Choose the right use cases

Build a strong data foundation

Whether you plan to fine tune an existing large language model (LLM), like OpenAI’s GPT-3.5 Turbo or Google’s LaMDA, or want to create your own foundation model, a good data strategy is essential. Good data architecture is critical for accelerated outcomes and for the age of AI to begin. Your eventual generative AI business solution will need to be able to access real-time data, curated document stores or vector databases, and work within established access controls and protocols to ensure regulations are met.

Laying a data fabric across your entire digital environment and establishing good data governance are the first steps. Generative AIs are only as good as their data; an erroneous label or instance of model drift can create hallucinations, mistakes or simply bad outputs. From there, you’ll have to make architectural adjustments specific to generative AI for business, such as designing prompt templates or prompt sequencing, retrieval augmented generation or model parameter tuning to get to the most relevant and optimal outcomes.

Data de-identification or anonymization to protect personally identifiable data or sensitive personal information data being consumed by the LLMs can be a key consideration from a data governance standpoint.

Data foundation hurdles aren’t insurmountable, but they’ll require a foundational data strategy to address them. Adding generative AI to your tech stack without data strategy changes is a recipe for cost overruns, compliance troubles and more.

Use Case: Data Foundation and SQL Lookups

In the case of a customer service department, ad-hoc requests for reports from structured query language (SQL) databases can be time-consuming for customer service teams. This is because hundreds of reports per month need to be completed, with many individual reports taking hours. Additionally, many customer service requests require agents to pore through large user guides and documentation to find a solution.

A generative AI customer service assistant trained on a business’s SQL database and user guides can translate natural language questions into SQL queries, run the queries, and then translate the results back into natural language for easy understandability. This process can also be applied to an entire library of user guides, enabling quick lookups and rapid responses to customer issues.

Leverage Large Language Model Operations (LLMOps)

With your data foundation established, you’ll next want to:

• Define which foundation models will be fit-for-purpose
• Set up prompt sequences or templates for your LLMs
• Set up vector stores
• Bring in responsible AI guardrails

The architecture, policies and procedures governing this body of work are referred to as LLMOps. It covers the entire generative AI lifecycle from compliance and security to drift and bias to prompt engineering and more.

LLMs need vast pools of data for training. One challenge is that if most of the training data is in English, the models may inadvertently assume Western points of view. Data bias in generative AI can show up in other ways, too: imagine using a customer service bot trained solely on northeastern US voices to field customer calls from the South of England. It may have difficulty interpreting regional dialects.

Another interpretation challenge may be context. “I’m locked out of my office” can mean one thing to a building security guard and another to a Microsoft customer service agent. A generative AI business solution may not be able to interpret intent.

Leveraging a LLMOps framework helps put AI to effective, optimal and responsible uses, both delivering for the bottom line and protecting an organization and its users from potential dangers.

Use Case: LLMOps, context, and customer feedback

A solid LLMOps strategy, supported by techniques such as retrieval augmented generation, can give generative AI the specific domain, cultural and situational context it needs to improve customer experiences.

Consider the example of an international airport in India, which gets thousands of reviews and comments each month on social media. Staff members had been manually managing and responding to feedback, which proved a hugely time-consuming process. The airport created an AI to scan all social media channels for customer comments and classify them by sentiment (for example, pleased versus unsatisfied) and intent (such as information-seeking versus recommending). The AI then generated appropriate responses for each, which were reviewed by managers before deployment. The solution affords the airport a new, efficient way to address customer pain points.

Manage for potential shadow AI  

In the early days of the cloud, it wasn’t uncommon for teams to use different platforms to store their documents and files. The practice, scaled across departments, created an intricate tangle of virtual platforms and associated security vulnerabilities.

While companies worked to streamline cloud storage, security practices and costs, employees still clung to preferred tools and platforms to do their work, creating shadow IT.

Companies can get ahead of a similar shadow AI situation by implementing responsible AI policies that pre-empt superfluous AI and mitigate security risks that could stem from unauthorized use. These policies should be flexible enough to encourage innovation within teams while establishing clear guardrails for compliance and ethical standards.

Cost management also applies in a straightforward weighing of options. One generative AI business solution’s architecture may achieve similar outcomes as another, but at a fraction of the cost. Different optimization techniques can also be utilized to get the best outcome with a much lower computing footprint.

Choose the right use cases

When considering use cases for generative AI in business, look for situations where its rapid ingestion, analysis and summarization capabilities will provide the most value to customers, employees or other users.

Avoid use cases that may introduce unfairness and bias, risk regulatory noncompliance or lead to reputational damage. Pick ones that can be implemented quickly, reliably and inexpensively. Start with prototypes that can be scaled out upon proof of concept. Know how you’ll measure ROI.

Here are some areas to look at to unleash the power of generative AI in business:

• Analyzing costs for optimization
• Automating rote tasks
• Examining customer support cases for trends
• Personalizing customer experiences

Use Case: transcription and summarization

Contact centers present a particularly ripe use case for generative AI with AI-generated summaries of customer support events. A typical customer support interaction happens, with the content and context of the interaction captured and analyzed by the AI.

The AI then generates a summary of the event, which the support agent reviews and verifies. This enables the next support agent to get up to speed and assist the user quickly, while reducing the time the previous support agent spends recapping the call.

The numbers back it up and show ample room to grow. In a study from the Stanford Digital Economy Laboratory and MIT Sloan School of Management, call centers saw a 13.8% increase in productivity when they implemented a generative AI assistant tool.³ And the market size for generative AI in customer service is predicted to be worth over $2.1 billion in 2032.⁴

The future of generative AI in business is only beginning to be written. Like any new technology, there are plenty of concerns and reservations, but with a strong foundation, a proactive strategy, cost optimization and the right use cases, the future can be bright.

Naveen Kamat is Vice President and CTO of Data and AI Services for Kyndryl. Dennis Perpetua is Vice President and CTO of Digital Workplace Services.

1 IDC Forecasts Spending on GenAI Solutions Will Reach $143 Billion in 2027 with a Five-Year Compound Annual Growth Rate of 73.3%. IDC. October 2023 

2 Economic potential of generative AI, McKinsey, June 2023IDC Forecasts Spending on GenAI Solutions Will Reach $143 Billion in 2027 with a Five-Year Compound Annual Growth Rate of 73.3%. IDC, October 2023

3 Measuring the Productivity Impact of Generative AI, National Bureau of Economic Research, June 2023

4 Generative AI in Customer Service Market worth around USD 2,103.0 Mn by 2032, Enterprise Apps Today, May 2023