When (Not) to Choose Google Apache Kafka for BigQuery?

Google announced its Apache Kafka for BigQuery cloud service at its conference Google Cloud Next 2024 in Las Vegas. Welcome to the data streaming club joining Amazon, Microsoft, IBM, Oracle, Confluent, and others. This blog post explores this new managed Kafka offering for GCP, reviews the current status of the data streaming landscape, and shares some criteria to evaluate when Kafka in general and Google Apache Kafka in particular should (not) be used.

(Originally posted on Kai Waehner’s blog: “When (Not) to Choose Google Apache Kafka for BigQuery?”… Join the data streaming community and stay informed about new blog posts by subscribing to my newsletter)

Welcome Google Apache Kafka to the Data Streaming Club

Better late than never… Google announced a brand new Apache Kafka cloud service for GCP at Google Cloud Next 2024. All other leading cloud providers already have one, including AWS, Azure, Oracle, IBM, and Alibaba. Various other software vendors provide Kafka services, including Confluent, Aiven, Redpanda, WarpStream, and many more. Most leverage the open source Kafka project as its core component, others re-implement the Kafka protocol.

Apache Kafka and Apache Flink dominate the open source data streaming ecosystem. Vendors and cloud…