Welcome: why was I told I'm not ready for AI? (0:08)
Moises Aguilar: Hello everyone, and welcome to our Road to AI webinar series. I have with me my co-host, Vivek Gopinathlal. How are you, sir?
Vivek Gopinathlal: I'm doing well. How are you?
Moises: Today's episode is on data standardization, or preparing the data for AI. The setup is that I'm playing the CIO of a company that has a cloud system, data integration, reports, databases, all kinds of things. Somebody came in and told me I'm not ready for AI. So that's the first question: why? What is going on with my systems that somebody came and told me, Moises, you're not ready?
The three standards your data needs (0:44)
Vivek: This is a typical question we hear from our clients too. They've been doing this data business for a while, and when they want to move toward AI, they get a report saying they're not ready. There are multiple reasons, but we'll start with the core, which is data.
Data has multiple layers. When you ask, "Do I have everything for the AI to process and give me a meaningful response?", you need certain standardizations. Standardization is very much required, and it can be structural, business, or semantic. These three are core. The final piece is consumption: how AI wants to consume all of this. If you have those three set up within your business, AI just needs an additional tool for the consumption layer, and you should be good.
Most of the time, what we miss is the business standardization. The structural standards come in through the engineering process, and the semantic ones come in through the reports, but they need to be kept in order. Only then is the data in a consumable format for AI to start responding.
The simplest path: start with one domain (2:38)
Moises: Same question as always: what's the simplest path? I don't want to spend a fortune, and I don't want to boil the ocean. I don't want to standardize my whole company. At a high level the concept makes sense, but when you get into the details, I don't know what I'm getting myself into.
Vivek: Most organizations already have a data engineering pipeline, and they've classified their data by certain domains. If that's missing, you start there. You just need to validate that the domains are categorized. If they are, it becomes easy.
Take wealth management as an example. Accounts are quite important. You have data coming from different portfolio services and different sources, and you classify it into an accounts domain, households at different levels, holdings, portfolios, each as a separate category. Pull the data into those structures, and picking one, accounts, is good enough. It's a standard one across industries. Pick that use case, pick the business value out of it, so you can also measure your success. That's key.
Picking a first use case: cross-sell recommendations (4:27)
Moises: Let's pick a different use case than the last episode, one that's more based on data. I want cross-sell. I'll pick an agent that has business value and requires data, but isn't business critical, so nobody's going to be hurt if it does something weird. I need cross-sell recommendations. It could be a retailer, wealth management, or insurance. I want to build an AI-based cross-sell recommendation engine. What's the simplest way? Can I limit by geography, or by business line? Is there a clever way to get something done in a normal amount of time? My systems aren't fantastic, but they aren't horrendous either. I don't have business logic in twenty-five places and holes everywhere. It's okay. So how do I do this fast?
Vivek: I'll go back to the layers: structural standardization, business standardization. When you qualify business standardization, that's where you define what a customer is, how you define revenue, where the revenue comes from, and what the different accounts, households, and products are.
The easiest use case would be a customer 360 for a specific product in a specific region. For a financial company, pick loans: a customer 360 for loan customers within a particular region. Bring it to a level where you can qualify, first, what a customer is, and second, how much revenue you can yield from that customer and what pathways bring you information about that customer. Once you understand the flow of data for that customer, bring those sources in, not every source, just the ones specific to this subset of your customers.
Most of the time, your data engineering pipeline already does a lot of the filtering, aggregation, and summarization. You build on top of it. You don't have to start from scratch fetching the data. Then you add context and your own value to it, so you can measure what you're getting out of it.
Start with a subset: every customer, fewest fields (7:38)
Moises: Let me see if I understand. You're saying bring in all the customers, but only the least amount of fields, the fewest attributes. What I can cut in budget or scope is the number of data sources or the amount of detail. Make a rough agent: the minimum amount of data, but all the clients.
Vivek: Yes. You don't have to boil the ocean, but you can pick a subset of data from these tables and sources and derive value from it. It's not going to be the whole picture. But be very clear about the quality of the response. It should not deviate from what the data says. We need to validate that with this subset and be sure it's not hallucinating or giving us something from the generative AI itself. It should answer specifically what the question asks and what the data says. If that's the case, start with that subset and then slowly scale to larger data volumes.
How do you stop AI from going bananas? (9:11)
Moises: I'm still trying to wrap my head around this. I want an agent that gives cross-sell recommendations. I bring the fewest fields I need to calculate them. I need all the transactions, whether I want to or not, to see what customers have. I need my products, to know what to cross-sell. And you mentioned verifying the outcome so it doesn't hallucinate. Isn't it as simple as checking that it's not inventing products? Or is it going to give me a crazy answer and I'll never know? How do I prepare my data to make sure it doesn't go bananas?
Vivek: Let's separate the topics. How you want your data to look for AI consumption is one thing. How we verify that the AI is responding properly is another. There are tools, services, and features you can build, and you can also create prompts so the AI answers to the point and doesn't wander off the topic.
We've done this for different clients. Whenever a response comes in, we play devil's advocate: "No, this data is wrong, can you revalidate?" It goes through a loop where the AI qualifies the response it has given and then returns the outcome. This has served us well. The AI revalidates itself and comes back with a final response, with traceability for how it arrived there. There is a cost to it, but it gives us confidence that this is closest to what we're looking for. Then you challenge the system again and see whether the response changes. From that you can arrive at a confidence score and project from there. That's how we validate the response.
How much data preparation is enough? (11:59)
Moises: Let me go back to my data, because somebody is saying I'm not ready. What is most important here: that it's complete, traceable, clean, standardized? I'm trying to assess how big the scope of preparing the data is. We said grab the customer and only a couple of fields, but I seem to need all the transactions and all the products. Is there a clever way, again, not to boil the ocean? Is there a minimum amount of preparation? I'm also thinking about architecture. Do I create a completely new environment just for AI consumption, or do I upgrade what I have and add layers of data preparation on top? Is it only for AI, or can I use it for reporting too? I'm trying to understand where I'm going to spend the money, and what business value I get back.
Vivek: Business value definitely comes first. Let's break your question down. You don't need a different system altogether. You already have a system and a warehouse. If you don't, then yes, you first have to build the data foundation layers, whether that's a medallion architecture, fact and dimension models, or whatever you prefer. Then you have your reporting layers. Once the core foundation is finished, you can consume directly from those layers without overwriting that data. You can read it, and from there build your AI sandbox layers and run them on the minimal data you have.
As for how you define whether your data is the best it can be: you can never fully qualify that. There's always something that would have made it better, another table to join. No one can say their data is 100% good. What you can do is make sure the quality is good. How? Primary keys with nulls are the simplest thing I can think of. Those basics you can't avoid. A few business rules you can't avoid either. If your business has only two products, loans and cards, and the data shows a different product, that's wrong. You can't contradict your own business with your data. Handle those data quality issues first.
Second is consistency. Yesterday I had a thousand customers, and today I have a hundred. No. There can be a drop, but make sure it's within a threshold, and the same for a peak. Sometimes there are good reasons: a retailer has sales around Thanksgiving, so the volume hikes. Those are valid scenarios, and you've identified and qualified them, so you have consistent data.
Third is timeliness. I can't say, "I got data that is three days old." Then your AI is giving you a three-day-old report. It might help with trend analysis, but not with decision-making. To take a decision, the data has to be point-in-time and the latest.
Make sure those three things are in place. They're the minimum. Then you build your AI consumption layers, which can be at different levels depending on how much a client wants to invest. For the simplest orchestration flows, use open source and readily available AI services such as LangChain and LangSmith. Orchestration is done, and a final bot can start answering questions on your data: checking the data, understanding what you're trying to do, and responding from there. Address those three factors first, and you'll also be able to answer for your value.
Moises: Give me the three key factors again.
Vivek: Data quality, data consistency, and timeliness of the data. Those three are key.
What can you do with it once it's ready? (18:06)
Moises: So those three factors, for a small subset of fields for my customer, lots of transactions, and lots of products. Now, what can I do with it?
Vivek: That's a typical question from the business. Most of the time, the business builds multiple products on top of the data, and that helps drive their product journey. Say the business wants to identify its competitors and what sales they're running right now. If I can give them good-quality data on the different competitors within their region and the sales happening there, and say it's the fall season, with school reopening and new needs for photography clients, then it becomes easy for them to come up with new products.
For that, we need the right data and AI that understands what's happening between last year and the previous year, how the dip or the trend is moving, and whether it's because of seasonal factors or because competitors are selling more. You pass that information on, and AI can tell you to change your sales strategy or campaign: rather than a 19% discount, go for 18%, or go for 35%. AI will recommend, but the decision still has to be taken by the business.
How much can you trust AI's reasoning? (20:33)
Moises: I've used AI a little, and it talks very decisively: "This is what you should do." Sometimes you go, hmm. So what kind of decisions can I trust? A computer can go through massive amounts of data that a human can't. The capacity to ingest, organize, classify, and sort is amazing. But the capacity to reason compared to a human isn't there yet. Even if I have the data ready, what can I trust this thing with today? We're at the end of 2026. How much can I trust what these tools tell me?
Vivek: You can't trust AI unless you trust the data. We all agree on that: garbage in, garbage out. Second, what level of tooling do you need to take that decision? I mean the journey from the data to the AI recommendation, and whatever tools are required in between. It's not necessarily AI itself. You don't need AI where everything is predefined rules, predefined agendas, and predefined business values. You can pick that up from existing software and tools.
If you want some context written on top of it, a generative AI can give you good context for a post or a marketing campaign. I can trust that piece. But when you want to see trends or identify a pattern, you generally go toward machine learning rather than AI by itself.
We can create our own models based on our data, because we know how it's distributed, and we know the marketing sense of it as humans. For trends and flows based on market analysis and other factors, I would still go with an ML model that tells me, "This is the pattern it's going in." I can run it multiple times, look at the patterns throughout the year during the peak seasons, and check that they make sense. Then I can give that information to AI for it to take its own understanding.
Now I've built a system where I have control. You build a system underneath that has all the factors your business needs, and then you put AI on top. That way you know AI is giving you an expected, deterministic response. No probabilistic systems.
Machine learning first, AI on top (24:36)
Moises: Let me rephrase. The machine learning step looks at all the data I have, finds the trends and expectations, and produces new data as a step in the process. Then I put AI on top of it, and AI uses my data plus whatever the machine learning produced, and together it gives me a better answer. Is that what you're saying?
Vivek: Exactly. This way you have control over your data and over the responses AI provides. That's the value.
Moises: Give me an example.
Vivek: The simplest example is customer sales. Sales keep happening, then all of a sudden there's a dip. If I give all the data to AI, it will say it's too much volume, or it will cost you a lot. Say sales suddenly dropped and during another period suddenly peaked. I want to figure out why. So I run the complete data, all the tables and features I want to analyze, and create an ML model to understand why it dipped and why it peaked. Maybe it's the weather, the season, or a competitor doing something different. It could be multiple factors from different sources. You create the ML layer and score it, and you see a trend and a pattern.
That's too much for us to consume as raw data, but when you give AI the ML layer's responses, how the trend goes and how the individual patterns work, it understands them easily. It breaks them down and helps you decide: probably due to this, probably due to that, with a good amount of confidence. Without that, it's guesswork. We don't want to go into a probabilistic mode. We want to go in a deterministic way.
Prepared data sets: smarter aggregation (27:41)
Moises: Let me rephrase again. If I take all my raw data and throw AI on top, AI has to deal with raw transactional data. If I'm clever enough to build machine learning models that produce conclusions, I get an extra step. The idea in my mind is a clever aggregation. Instead of just rolling numbers up as we used to with data marts or aggregate tables, it's a more logical, machine learning aggregation. If I prepare these data sets neatly enough, AI consuming that prepared data raises the chances of it working well. That's data preparation too, right?
Vivek: Absolutely right. You put it nicely. There's still a gap between that data and the AI, but this level of data is good enough for AI to start answering and understanding the patterns. All LLMs come with their own general knowledge, and some of it is business-specific. If you want it to be specific to your enterprise, build the semantic standardization. That allows you to communicate between the layers very well, and the responses will be very focused on your business products.
Raising your level of trust in AI (29:41)
Moises: So now I have a better understanding of how to prepare my data. But I've been using AI for a while, and there are things I can trust it with and things I can't. What can I do to raise the level of trust in the AI itself, from the data perspective and beyond? How can I get the best out of its reasoning and understanding?
Vivek: Trust in the data is what we need. If you trust your data, you can trust your AI's response. It's garbage in, garbage out. The reasoning can happen at different levels, and there are different tools for these purposes. If you have a standard set of campaigns and a business going in a steady direction, there are software tools readily available, and you don't need AI. If you need help with a campaign system, you can use generative AI to help you write things, and there you can have a different level of validation, because generative AI does really well in that area.
To make the right decisions on your data, you need AI tools that connect all the information in your data and then have AI respond on top of it. For your AI system to respond deterministically, not probabilistically, you need to qualify the tools and how they interact with the AI. It could be a simple API tool connecting to the right sources, or a machine learning tool that identifies patterns and trends in your data.
The response won't always be 100% accurate, but you gain confidence, and you can recommend to a level close to 95% or 99% accuracy. Then it's easy for a human to take a decision on top of it. Trust comes when you have the right tools. Machine learning can identify trends, understand where your data lies, and spot outliers or data that's getting into outlier territory. You can't feed all the data to AI and ask it to understand it. It's going to cost you a lot, and it will start hallucinating after a point. Understand the different tools that are available and how to use them so AI can answer better or recommend better responses.
Moises: We will get deep into the semantic layer in the next episode. For now, thank you very much, sir. Thank you for all your information, it was enlightening. Thank you, everyone, for watching, and I'll see you in the next episode.
Vivek: Awesome. I appreciate you. Thank you.