Best Kafka Summit Videos

The following talks, with video recordings and slides available, achieved the best ratings by the community at the Kafka Summit conferences from 2018 onwards. Thanks to all the speakers for their hard work!

TABLE OF CONTENTS

• Kafka Internals and Fundamentals
• Applications and Use Cases
• Architecture and Patterns
• Data Pipelines
• Kafka Operations

Kafka Internals and Fundamentals

• Making Kafka Cloud Native, Jay Kreps (Confluent), KS EU 2021
• How ksqlDB works, Michael Drogalis (Confluent), KS EU 2021
• Understanding Kafka Produce and Fetch API Calls for High Throughput Applications, Jason Gustafson (Confluent), KS EU 2021
• A Kafkaesque Raft Protocol, Mik Kocikowski (Cloudflare), KS EU 2021
• Kafka ♥ Cloud (Keynote), Jay Kreps (Confluent), KS 2020
• Trade-offs in Distributed Systems Design: Is Kafka The Best?, Ben Stopford & Michael G. Noll (Confluent), KS 2020
• The Flux Capacitor of Kafka Streams and ksqlDB, Matthias J. Sax (Confluent), KS 2020
• Crossing the Streams: the New Streaming Foreign-Key Join Feature in Kafka Streams, John Roesler (Confluent), KS 2020
• Welcome to Kafka; We're Glad You're Here, Dave Klein (Centene), KS 2020
• Kafka's New Architecture (Keynote), Gwen Shapira (Confluent), KS 2020
• Getting Started with Apache Kafka – a Contributor's Journey, Israel Ekpo (Microsoft) & Matthias J. Sax (Confluent) & Nikolay Izhikov (Sberbank), KS 2020
• Kafka Needs no (Zoo)Keeper (abstract), Jason Gustafson & Colin McCabe (Confluent), SFO 2019
• Why Stop the World When you Can Change it? Design and Implementation of Incremental Cooperative Rebalancing (abstract), Konstantine Karantasis (Confluent), SFO 2019
• Kafka 102: Streams and Tables All the Way Down (abstract), Michael G. Noll (Confluent), SFO 2019
• What’s the time? …and why? (abstract), Matthias J. Sax (Confluent), SFO 2019
• Zen and the Art of Streaming Joins: The What, When and Why (abstract), Nick Dearden (Confluent), NYC 2019
• Exactly Once Semantics Revisited (abstract), Jason Gustafson (Confluent), NYC 2019
• Performance Analysis and Optimizations for Kafka Streams Applications (abstract), Guozhang Wang (Confluent), LON 2019
• Everything You Always Wanted to Know About Kafka’s Rebalance Protocol but Were Afraid to Ask (abstract), Matthias J. Sax (Confluent), LON 2019
• Reliable Message Delivery with Apache Kafka (abstract), Ying Zheng & Xiaobing Li (Uber), SFO 2018
• Hardening Kafka Replication (abstract), Jason Gustafson (Confluent), SFO 2018
• Don’t Repeat Yourself: Introducing Exactly-Once Semantics in Apache Kafka (abstract), Matthias J. Sax (Confluent), LON 2018

Applications and Use Cases

• Should You Read Kafka as a Stream or in Batch? Should You Even Care?, Ido Nadler (Nielsen) & Opher Dubrovsky (Nielsen)
• Scaling a Core Banking Engine Using Apache Kafka, Peter Dudbridge (Thought Machine), KS APAC 2021
• Scaling an Event-Driven Architecture with IBM and Confluent, Antony Amanse (IBM) & Anton McConville (IBM), KS EU 2021
• Development of Dynamic Pricing for Tours Using Real-time Data Feeds, Mourad Benabdelkerim (FREE NOW), KS EU 2021
• Building Event Streaming Applications with Pac-Man, Ricardo Ferreira (Confluent), KS 2020
• Apache Kafka, Tiered Storage and TensorFlow for Streaming Machine Learning without a Data Lake, Kai Waehner (Confluent), KS 2020
• KSQL-ops! Running ksqlDB in the Wild, Simon Aubury (ThoughtWorks), KS 2020
• Streaming Towards Our Quantum Future, David Elbert & Tyrel McQueen (Johns Hopkins University), KS 2020
• KafkaConsumer – Decoupling Consumption and Processing for Better Resource Utilization, Igor Buzatović (Inovativni trendovi d.o.o), KS 2020
• Can Kafka Handle a Lyft Ride?, Andrey Falko & Can Cecen (Lyft), KS 2020
• Flattening the Curve with Kafka, Rishi Tarar (Northrop Grumman Corp.), KS 2020
• Risk Management in Retail with Stream Processing, Daniel Jagielski (Virtuslab/Tesco), KS 2020
• 0-60: Tesla’s Streaming Data Platform (abstract), Jesse Yates (Tesla), SFO 2019
• Eventing Things – A Netflix Original! (abstract), Nitin Sharma (Netflix), SFO 2019
• Mission-Critical, Real-Time Fault-Detection for NASA’s Deep Space Network using Apache Kafka (abstract), Rishi Verma (NASA Jet Propulsion Laboratory), SFO 2019
• Using Kafka to Discover Events Hidden in your Database (abstract), Anna McDonald (SAS Institute), SFO 2019
• Building an Enterprise Eventing Framework (abstract), Bryan Zelle (Centene) & Neil Buesing (Object Partners, Inc), SFO 2019
• ksqlDB Performance Tuning for Fun and Profit (abstract), Nick Dearden (Confluent), SFO 2019
• Discovering Drugs with Kafka Streams (abstract), Ben Mabey & Scott Nielsen (Recursion Pharmaceutical), SFO 2019
• Being an Apache Kafka Developer Hero in the World of Cloud (abstract), Ricardo Ferreira (Confluent), SFO 2019
• Streaming Apps and Poison Pills: handle the unexpected with Kafka Streams (abstract), Loic Divad (Xebia France), SFO 2019
• Scaling for India's Cricket Hungry Population (abstract), Bhavesh Raheja & Namit Mahuvakar (Hotstar), SFO 2019
• Leveraging Services in Stream Processor Apps at Ticketmaster (abstract), Derek Cline (Ticketmaster), SFO 2019
• Stream Processing with the Spring Framework (Like You’ve Never Seen It Before) (abstract), Josh Long (Pivotal), Tim Berglund (Confluent), NYC 2019
• How To Use Apache Kafka and Druid to Tame Your Router Data (abstract), Rachel Pedreschi (Imply Data), NYC 2019
• Kafka Connect and ksqlDB: Useful Tools in Migrating from a Legacy System to Kafka Streams (abstract), Alex Leung & Danica Fine (Bloomberg L.P.), NYC 2019
• Building Serverless Apps with Kafka (abstract), Dale Lane (IBM), LON 2019
• Introducing Events and Stream Processing into Nationwide Building Society (abstract), Robert Jackson & Pete Cracknell (Nationwide Building Society), LON 2019
• The Exciting Frontier of Custom ksqlDB Functions (abstract), Mitch Seymour (Mailchimp), LON 2019
• Using Location Data to Showcase Keys, Windows, and Joins in Kafka Streams DSL and ksqlDB (abstract), Neil Buesing (Object Partners, Inc), LON 2019
• ksqlDB in Practice (abstract), Almog Gavra (Confluent), LON 2019
• Industry-ready NLP Service Framework Based on Kafka (abstract), Bernhard Waltl & Georg Bonczek (BMW Group), LON 2019
• Data Streaming Ecosystem Management at Booking.com (abstract), Alex Mironov (Booking.com), SFO 2018
• Kafka in the Enterprise—A Two-Year Journey to Build a Data Streaming Platform from Scratch (abstract), Benny Lee & Christopher Arthur (Commonwealth Bank of Australia), SFO 2018
• Life is a Stream of Events (abstract), Bjørn Kvernstuen & Tommy Jocumsen (Norwegian Directorate for Work and Welfare), SFO 2018
• Kafka Multi-Tenancy—160 Billion Daily Messages on One Shared Cluster at LINE (abstract), Yuto Kawamura (LINE Corporation), SFO 2018
• Matching the Scale at Tinder with Kafka (abstract), Krunal Vora (Tinder), SFO 2018
• Building an Enterprise Streaming Platform at Capital One, Chris D'Agostino (Capital One), SFO 2018
• More Data, More Problems: Scaling Kafka-Mirroring Pipelines at LinkedIn (abstract), Celia Kung (LinkedIn), SFO 2018
• From Propeller to Jet: Upgrading Your Engines Mid-Flight (abstract), Christopher Morris (PagerDuty), SFO 2018
• A Visual Approach to Understanding Streaming SQL (abstract), Shant Hovespian (Arcadia Data), SFO 2018
• Digital Transformation in Healthcare with Kafka—Building a Low Latency Data Pipeline (abstract), Dmitry Milman & Ankur Kaneria (Express Scripts), SFO 2018
• Building Pinterest Real-Time Ads Platform Using Kafka Streams (abstract), Liquan Pei & Boyang Chen (Pinterest), SFO 2018
• Taking Stateful Stream Processing to the Next Level with Kafka and Flink (abstract), Stephan Ewen (Ververica), LON 2018
• Taming Billions of Metrics and Logs at Scale: Two Years with Kafka as a Central Data Hub for Monitoring at CERN (abstract), Luca Magnoni (CERN), LON 2018
• ksqlDB 201: A Deep Dive into Query Processing (abstract), Hojjat Jafarpour (Confluent), LON 2018
• The Evolution of Kafka at ING Bank (abstract), Timor Timuri & Richard Bras (ING), LON 2018

Architecture and Patterns

• Mistakes - I’ve Made a Few. Blunders in Event-driven Architecture, Simon Aubury (ThoughtWorks Australia), KS APAC 2021
• Better CQRS with ksqlDB, Anshuman Mukherjee (Airwallex), KS APAC 2021
• Sharing Data Among Microservices: How Change Data Capture with Kafka Connect Came to Our Rescue, Ali Nazemian (Brolly) & Milad Vahood (Brolly), KS APAC 2021
• Apache Kafka and the Data Mesh, Ben Stopford (Confluent) & Michael Noll (Confluent), KS EU 2021
• How to Build the Data Mesh Foundation: A Principled Approach, Zhamak Dehghani (ThoughtWorks), KS EU 2021
• Getting up to Speed with MirrorMaker 2, Mickael Maison (IBM) & Ryanne Dolan (Twitter), KS EU 2021
• Reacting to an Event-Driven World, Kate Stanley & Grace Jansen (IBM), KS 2020
• A Tale of Two Data Centers: Kafka Streams Resiliency, Anna McDonald (Confluent), KS 2020
• Kafka as your Data Lake – is it Feasible?, Guido Schmutz (Trivadis), KS 2020
• GDPR Compliance: Transparent Handing of Personally Identifiable Information in Event-Driven Systems, Masih Derkani (SolarWinds), KS 2020
• Synchronous Commands over Apache Kafka, Neil Buesing (Object Partners, Inc), KS 2020
• Hybrid Kafka, Taking Real-time Analytics to the Business, Cody Irwin (Google Cloud) & Josh Treichel (Confluent) & Jeff Ferguson (Confluent), KS 2020
• I Don't Always Test My Streams, But When I Do, I Do it in Production, Viktor Gamov (Confluent), KS 2020
• Building Information Systems using Event Modeling, Bobby Calderwood (Evident Systems), KS 2020
• Building Event Driven Architectures with Kafka and Cloud Events (abstract), Dan Rosanova (Microsoft), SFO 2019
• Event Sourcing, Stream Processing, and Serverless, Ben Stopford (Confluent), SFO 2019
• Shattering The Monolith(s) (abstract), Martin Kess (Namely), SFO 2019
• Hard Truths About Streaming and Eventing (abstract), Dan Rosanova (Microsoft), NYC 2019
• Complex Event Flows in Distributed Systems (abstract), Bernd Ruecker (Camunda), NYC 2019
• Cost Effectively and Reliably Aggregating Billions of Messages Per Day Using Apache Kafka (abstract), Chunky Gupta & Osman Sarood (Mist Systems), NYC 2019
• Tracing for Kafka-Based Applications: Making Sense of Your Event-Driven Dataflows (abstract), Jorge Esteban Quilcate Otoya (SYSCO AS), NYC 2019
• From a Million to a Trillion Events Per Day: Stream Processing in Ludicrous Mode (abstract), Shrijeet Paliwal (Tesla) NYC 2019
• The Migration to Event-Driven Microservices (abstract), Adam Bellemare (Flipp), NYC 2019
• Talking Traffic: Data in the Driver’s Seat (abstract), Dominique Chanet (Klarrio), LON 2019
• Privacy Engineering for the World of Kafka (abstract), Alexander Cook (Privitar), LON 2019
• Riddles of Streaming – Code Puzzlers for Fun & Profit (abstract), Nick Dearden (Confluent), LON 2019
• Is Kafka a Database?, Martin Kleppmann (University of Cambridge), LON 2019
• Event-Driven Workflow: Monitoring and Orchestrating Your Microservices Landscape with Kafka and Zeebe (abstract), Mike Winters & Sebastian Menski (Camunda), LON 2019
• Handling GDPR with Apache Kafka: How to Comply Without Freaking Out? (abstract), David Jacot (Independent, formally Swisscom), LON 2019
• One Key to Rule them All (abstract), Richard Noble & Francesco Nobilia (Babylon Health), LON 2019
• How To Use Kafka and Druid to Tame Your Router Data (abstract), Rachel Pedreschi And Eric Graham (Imply Data), LON 2019
• Apache Kafka and Event-Oriented Architecture, Jay Kreps (Confluent), SFO 2018
• Bringing Streaming Data To The Masses: Lowering The “Cost Of Admission” For Your Streaming Data Platform (abstract), Bob Lehmann (Bayer), SFO 2018
• Breaking Down a SQL Monolith with Change Tracking, Kafka and KStreams/ksqlDB (abstract Wanny Morellato (SAP Concur), SFO 2018
• Kafka as an Eventing System to Replatform a Monolith into Microservices (abstract), Madhulika Tripathi (Intuit), SFO 2018
• Experimentation Using Event-based Systems, Martin Fowler & Toby Clemson (ThoughtWorks), LON 2018

Data Pipelines

• On Track with Apache Kafka: Building a Streaming ETL Solution with Rail Data, Robin Moffatt (Confluent), KS APAC 2021
• Apache Kafka and ksqlDB in Action: Let's Build a Streaming Data Pipeline!, Robin Moffat (Confluent), KS 2020
• Change Data Capture Pipelines with Debezium and Kafka Streams, Gunnar Morling (Red Hat), KS 2020
• Bravo Six, Going Realtime. Transitioning Activision Data Pipeline to Streaming, Yaroslav Tkachenko (Activision), KS 2020
• How Kroger Embraced a "Schema First" Philosophy in Building Real-time Data Pipelines, Rob Hoeting, Rob Hammonds & Lauren McDonald (Kroger), SFO 2019
• Building Stream Processing Applications with Apache Kafka Using ksqlDB Robin Moffat (Confluent), SFO 2019
• Kafka Connect: Operational Lessons Learned from the Trenches (abstract), Elizabeth Bennett (Confluent), SFO 2019
• Solutions for bi-directional integration between Oracle RDBMS and Apache Kafka (abstract), Guido Schmutz (Trivadis), SFO 2019
• Streaming Ingestion of Logging Events at Airbnb (abstract), Hao Wang (Airbnb), NYC 2019
• No More Silos: Integrating Databases into Apache Kafka (abstract), Robin Moffatt (Confluent), NYC 2019
• Lessons Learned Building a Connector Using Kafka Connect (abstract), Katherine Stanley & Andrew Schofield (IBM UK), NYC 2019
• Building Scalable and Extendable Data Pipeline for Call of Duty Games (abstract), Yaroslav Tkachenko (Activision), NYC 2019
• From Zero to Hero with Kafka Connect (abstract), Robin Moffatt (Confluent), LON 2019
• Change Data Streaming Patterns For Microservices With Debezium (abstract), Gunnar Morling (Red Hat), LON 2019
• Beyond Messaging: Enterprise-Scale, Multi-Cloud Intelligent Routing (abstract), Anand Phatak (Adobe), SFO 2018
• You Must Construct Additional Pipelines: Pub-Sub on Kafka at Blizzard (abstract), Stephen Parente & Jeff Field (Blizzard), SFO 2018
• So You Want to Write a Connector? (abstract), Randall Hauch (Confluent), SFO 2018

Kafka Operations

• Mind the App: How to Monitor Your Kafka Streams Applications, Simon Aubury (ThoughtWorks), KS EU 2021
• Encrypting Kafka Messages at Rest to Secure Applications, Robert Barnes (HashiCorp), KS EU 2021
• Everything You Ever Needed to Know About Kafka on Kubernetes but Were Afraid to Ask, Jakob Scholz (Red Hat), KS EU 2021
• Fully-Managed, Multi-Tenant Kafka Clusters: Tips, Tricks, and Tools, Christopher Beard (Bloomberg), KS 2020
• Overcoming the Perils of Kafka Secret Sprawl, Tejal Adsul (Confluent), KS 2020
• Kafka Lag Monitoring For Human Beings, Elad Leev (AppsFlyer), KS 2020
• Sharing is Caring: Toward Creating Self-tuning Multi-tenant Kafka, Anna Povzner (Confluent), KS 2020
• Kafka on Kubernetes: Keeping It Simple, Nikki Thean (Etsy), SFO 2019
• Please Upgrade Apache Kafka. Now! Gwen Shapira (Confluent), SFO 2019
• Tackling Kafka, with a Small Team, Jaren Glover (Robinhood), SFO 2019
• Secure Kafka at scale in true multi-tenant environment (abstract), Vishnu Balusu & Ashok Kadambala (JP Morgan Chase), SFO 2019
• Running large scale Kafka upgrades at Yelp (abstract), Manpreet Singh (Yelp), SFO 2019
• Production Ready Kafka on Kubernetes (abstract), Devandra Tagare (Lyft), SFO 2019
• Kafka Cluster Federation at Uber (abstract), Yupeng Fui & Xiaoman Dong (Uber), SFO 2019
• Experiences Operating Apache Kafka® at Scale (abstract), Noa Resare (Apple), NYC 2019
• Kafka Pluggable Authorization for Enterprise Security (abstract), Anna Kepler (Viasat), NYC 2019
• Kafka on Kubernetes: Does it really have to be “The Hard Way”? (abstract), Viktor Gamov and Michael Ng (Confluent), NYC 2019
• Flexible Authentication Strategies with SASL/OAUTHBEARER (abstract), Michael Kaminski (The New York Times) & Ron Dagostino (State Street Corp), NYC 2019
• Bulletproof Apache Kafka with Fault Tree Analysis (abstract), Andrey Falko (Lyft), NYC 2019
• The Foundations of Multi-DC Kafka (abstract), Jakub Korab (Confluent), LON 2019
• Show Me Kafka Tools That Will Increase My Productivity (abstract), Stephane Maarek (DataCumulus)
• Running Kafka in Kubernetes: A Practical Guide (abstract), Katherine Stanley (IBM UK), LON 2019
• Running Production Kafka Clusters in Kubernetes (abstract), Balthazar Rouberol (Datadog), LON 2019
• Don’t Be Scared: Multi-Tenant Cluster Support at Scale (abstract), Kelly Attaway & Cabe Waldrop (Pandora Media), LON 2019
• Kafka on ZFS: Better Living Through Filesystems (abstract), Hugh O'Brien (Jet.com), SFO 2018
• Kafka Security 101 and Real-World Tips (abstract), Stephane Maarek (DataCumulus), SFO 2018
• End-to-End Security with Confluent Platform (abstract), Vahid Fereydouny (Confluent), SFO 2018
• Robust Operations of Kafka Streams (abstract), Bill Bejeck (Confluent), SFO 2018
• Deploying Kafka Streams Applications with Docker and Kubernetes (abstract), Gwen Shapira & Matthias J. Sax (Confluent), SFO 2018
• URP? Excuse You! The Three Metrics You Have to Know (abstract), Todd Palino (LinkedIn), SFO 2018
• Monitor Kafka Like a Pro (abstract), Gwen Shapira & Xavier Léauté (Confluent), LON 2018