Snowflake and Databricks Summits 2024

Keynotes of the Snowflake and Databricks Summits in 2024

Recap of the key themes and announcments from the Snowflake and Databricks summits in San Francisco.

Snowflake and Databricks Summits 2024: A Tale of Two Data Giants

In back-to-back weeks in early June, the data world converged on San Francisco’s Moscone Center for the Snowflake and Databricks Summits. These events showcased the rapid evolution of data platforms and the fierce competition between two industry leaders. Let’s dive into the key themes and announcements from each summit.

Snowflake Summit: Unifying the Data Ecosystem

Snowflake’s vision centered around creating an all-encompassing enterprise data platform where various components integrate seamlessly. Here are the major highlights:

  1. AI Integration: Snowflake introduced built-in Large Language Models (LLMs) while also offering flexibility to access external models. This move positions Snowflake as a key player in the AI-driven data landscape.

  2. Hardware Partnerships: Snowflake announced a collaboration with NVIDIA, aiming to leverage specialized hardware for enhanced performance in AI and data processing tasks.

  3. Expanding Database Offerings: Relational AI, a graph database, is available as a Snowflake native app. Snowflake is pushing into more diverse data structures and query capabilities.

  4. Polaris and Iceberg: Snowflake announced Polaris, an open source catalog for Iceberg, which will be available in 90 days. The increased support for the Iceberg table format demonstrates Snowflake’s commitment to open standards and interoperability.

  5. Geospatial Capabilities: Significant advancements in geospatial data handling were showcased, opening up new use cases for location-based analytics.

  6. Data Sharing and Native Apps: Snowflake Live Sharing and Native Apps underscore the platform’s focus on collaborative data ecosystems and custom application development.

Databricks Summit: Unifying the Data Lakehouse

Databricks’ event was marked by strategic moves to solidify its position in the data-lakehouse market. Key themes included:

  1. Tabular Acquisition and Open-sourcing the Unity Catalog: The billion-dollar acquisition of Tabular, the company behind Apache Iceberg, signals Databricks’ commitment to open table formats and interoperability. In continuing their rivalry with Snowflake, Databricks strategically timed their acquisition while the Snowflake conference was going on. With the acquisition, Databricks is encouraging enterprises to embrace data lakes without worrying about the details of the table format. Open-sourcing the Unity Catalog reflects Databricks’ dedication to transparency and community-driven development.

  2. LLM Integration: Similar to Snowflake, Databricks has its own LLM models but also integrates with other leading models.

  3. AI and BI Integration: Databricks announced an innovative AI/BI product, which is aiming to reimagine business intelligenceI. Having a tightly integrated chatbot, which is constantly learning, next to the BI dashboard seems like a compelling product.

  4. Data Sharing and Privacy: Delta Sharing and Clean Room features address growing needs for secure data collaboration across organizations.

  5. Language Support: Elevating Python to a first-class citizen in Spark, alongside the existing Scala support, caters to a broader developer base.

  6. Performance Enhancements: The Photon engine represents Databricks’ continued focus on query performance and efficiency.

  7. Serverless and Simplification: Emphasis on serverless architectures aims to reduce complexity for users.

  8. Agentic Design: Introduction of agent-based systems for specific tasks, leveraging prompting and chain-of-thought methodologies.

  1. Convergence of Platforms: Many enterprises are using both Databricks and Snowflake, indicating a trend towards multi-platform strategies. Iceberg is emerging as a key technology for maintaining a single copy of data and facilitating communication between Databricks and Snowflake environments.

  2. Focus on AI and LLMs: Both companies are heavily investing in Generative AI capabilities. They don’t just offer in-built vector databases and foundational models, but also integrate with other leading players.

  3. Nvidia is everywhere: Nvidia hardware will be available on both platforms, powering AI workloads.

  4. Open Standards and Interoperability: There’s a clear shift towards open formats and interoperability, as evidenced by Snowflake’s adoption of Iceberg and Databricks’ acquisition of Tabular.

  5. Simplification and Accessibility: Both platforms are striving to make complex data operations more accessible through serverless architectures and simplified interfaces.

  6. Ecosystem Expansion: Native apps and marketplace features from both companies indicate a push towards creating broader data ecosystems.

  7. Cost-Effective Solutions: While the conferences were going on, DuckDB, the fast in-process analytical DB announced its v1, signifying its readiness for production use. At the same time, MotherDuck, the company offering a cloud data warehouse based on DuckDB, announced its General Availability. It promises data warehousing capabilities at a fraction of the cost. .

These two data summits not only showcased the latest innovations from Snowflake and Databricks but also highlighted the dynamic and competitive nature of the data platform market. As these giants continue to evolve and compete, the real winners are the enterprises and data professionals who benefit from increasingly powerful, flexible, and interoperable data solutions.