Want to Scale AI? You’ll Need the Right Data Architecture Strategy

Without the right foundation, AI can’t support your business efficiently. Here’s what you need to ensure that doesn’t happen.

Many businesses are getting serious about deploying AI in their everyday work. With or without realizing it, they are depending on a modern data architecture to make AI efficient and relevant. Even so, data architecture strategies can all too often be overlooked; data storage and computing power has gotten much less expensive, which also makes it seem less important than it really is. But if you don’t get the data architecture right, it will be harder to get the AI results you want.

So, let’s look at what the modern data stack does and what we need to do to build a reliable one. As usual, we won’t go into the details; this will depend on your business. But we will cover the important concepts so you can make an informed decision.

The Modern Data Stack Meets Business Needs

A tech stack is a collection of the tools and technologies that businesses use to support their technical endeavors; a data stack is the tools, technologies, techniques, and services businesses use to support their data endeavors. A modern data stack should support all data work and uses cases, including: 

• Data processing and transformation 
• Data science and analysis 
• Pushing data to Business Intelligence (BI) and data visualization tools, user dashboards, and other consumption points 

Along the way, the tech stack should also support data scientists and data analysts in their work and allow business users (i.e. who are not IT or data professionals) to run the tasks they need in their work. As a modern stack, it should also use automation and orchestration to work independently and effectively. 

In short, a modern data stack needs to allow all users to do their work on it in a seamless and efficient way. This is a goal that many legacy data architectures simply aren’t designed to meet. So let’s talk about three critical things to consider when you’re creating a modern data architecture. 

3 Keys to a Scalable Data Architecture

1. Centralize with Care

Data centralization has been hailed as the cure for many data ills – but, like any curative, it can be overused and misapplied. The truth is that many companies have tried to over-centralize their data. And sometimes, they’ve tried and failed to do this repeatedly. 

 The rationale can range from wanting to quickly get results and value from AI projects to putting the wrong parts of the equation first. And because data centralization is so heavily promoted, it can be all too easy to adopt a “centralize at all costs and all events” mindset. But data centralization isn’t the only path to a good data architecture.  

 The data mesh – where data resides with various teams and the business itself assumes the responsibility to make it available to other areas as needed – is one such solution. It’s gotten a lot of attention recently, but it proves that other possibilities than a monolithic data ‘single source’ exist. The takeaway is that data centralization need not be the only solution you recognize; be flexible and see what else might work with your company’s data and organizational structure. 

2. Factor In the Human Element (and Lean Into Cross-Skill Teams)

In a perfect world, all you’d need to do would be to deploy some AI solution and it would automatically do everything necessary for you to start collecting data-backed insights. In reality, generating insights from your data is a ton of work. And it requires a truly varied skillset. Just think of all that has to be done both technically (sourcing, cleaning, and transforming the data, plus designing systems to transport it through the company, plus creating the analytical models, plus building the delivery systems, plus training and maintaining the AI systems, etc.) and from a business perspective (i.e. creating business rules, setting context, and validating results).  

To solve this, you need cross-discipline teams – and input from multiple teams and stakeholders. It can be tempting to focus exclusively on the technical aspects, but don’t underestimate the amount of human intelligence and hard work that needs to go into a successful AI initiative. We can’t automate everything; you need to factor the human element into your data stack. 

3. Keep Business Objectives and Tech Realities in Mind

We’ve all heard about AI projects that work brilliantly in a technical sense but totally miss their business objectives. In the excitement of setting up this new thing, we may forget important conversations around a well-defined business case and considering all use cases and how we will get actual value from our shiny new AI tools. The preventative for this is simple: design your data architecture with your business needs and realities in mind. Otherwise, users may try to do things that the data was never designed to accomplish.  

To avoid this (and to avoid making users find workarounds for something that should have been included in the official design strategy), figure out what kinds of data your organization has and how it will create value for you. What will business users need to do with it? What will technical users want it to do? Isolate the “why” behind these questions and ensure you have a clear idea of the motivations behind collecting, storing, and working with all this data. 

Make Sure Your Data Stack Will Support Your AI Needs

In conclusion, make sure you have a strong idea of your organization’s business and technical goals before you dig into data architecture. Be especially clear on how data and AI will be used currently as well as how they might grow in the future. When you start from supporting your business needs, you’re more likely to develop a data stack that will scale, support, and ultimately derive value from AI in your organization. 

Authored by: Dr. Anil Kaul, CEO of Absolutdata and Chief AI Officer at Infogain and Anil Joshi, Vice President at Absolutdata -an Infogain company

Subscribe