Data-powered AI revolution: Enhancing enterprise with Databricks

INFRA

As artificial intelligence continues to take the enterprise world by storm thanks to evolving innovations, such as ChatGPT, getting the best out of data-powered AI is a critical.

In today’s enterprise technology ecosystem, Databricks Inc. advocates for the integration of data-driven AI within a unified stack to achieve optimal performance, according to Craig Wiley (pictured), senior director of product for machine learning and AI at Databricks.

“Within the space of AI, this really comes down to … data plus AI,” he said. “If you look at, for example, any of the cloud, the hyperscaler AI platforms, I think what you find is that there are these distinct AI platforms separate from all of the rest of the tooling. They can be connected to all the other tooling through clear APIs, but Databricks has really done this differently. Databricks has said, ‘Hey, what if data and AI were managed completely within the single stack?”’

Wiley spoke with theCUBE Research principal analyst Rob Strechay at the “Supercloud 6: AI Innovation” event, during an exclusive broadcast on theCUBE, SiliconANGLE Media’s livestreaming studio. They discussed the importance of data-powered AI and how Databricks makes this a reality.

Through the data-powered AI lens

Since data is the backbone of AI models, having optimal lineage, governance and security is critical. As a result, Databricks increases developer velocity through data-powered AI based on the idea of unifying not only the data, but also the assets that are created from it, according to Wiley.

“Using our feature store and training a model, your data can actually just inherit the governance of the data in the feature store and then your model is fully governed,” he said. “Whoever had access to the data, they’re the only ones who have access to that model unless you grant others access. Having one lineage graph and one governance capability across all of these data assets is really a massive opportunity in this space.”

As generative AI continues to take center stage in the enterprise field, Databricks is enabling data teams to go a notch higher through its feature store, which is co-designed with an MLOps framework and data platform for better forecasts. The company’s Vector Search is also a part of Databricks’ Data Intelligence Platform, since it allows developers to enhance the accuracy of their gen AI applications, according to Wiley.

“I think with gen AI, we have a new opportunity to learn how is it that we drive accuracy or how is it that we drive fit between our model and the use case,” he said. “We are seeing this idea … of merging the data catalog and the feature store or the vector store, the ability for people to access the data so that data scientists are no longer using these separate feature store systems where all the data has to be copied and pushed over into these.”

Building data pipelines is a thing of the past thanks to Databricks’ feature store, Wiley added. This is made a reality by merging AI and data, since various capabilities, such as monitoring, are enhanced for better results.

“Let’s say you’re using our model monitoring capability and you’re looking at your models for skew and drift,” he said. “With an AI system that’s completely connected to the data system, we can sit there and say, ‘Hey, your model’s drifting, and we think it’s because feature No. 4 is broken, and the reason we think feature No. 4 is broken is there’s more nulls in that table than there ever have been before, and we actually know the load job that put those nulls in.’”

Here’s the complete video interview, part of SiliconANGLE’s and theCUBE Research’s coverage of the “Supercloud 6: AI Innovation” event:

Photo: SiliconANGLE