Designing Data Infrastructure in the Age of Generative AI

Developing powerful Al tooling has been our theme of the year, with agents and foundational models picking up steam across the board. Therein still lies the question though: how do we serve data for agents to work effectively?

What sort of interfaces and service mesh infrastructures will be required?

What about at enterprise scale?

What even is context?

In this talk we discuss the current big data landscape, challenges to data platforming for Al, and the shifting importance of open table formats, catalogs, and engines as means for effective, governed Al-development. In this talk we use the open source technologies such as Apache Spark, Unity Catalog OSS, and Apache Iceberg as key components of such reference architecture.