From Zero to Hero with Kafka Connect

Integrating Apache Kafka with other systems in a reliable and scalable way is often a key part of a streaming platform. Fortunately, Apache Kafka includes the Connect API that enables streaming integration both in and out of Kafka. Like any technology, understanding its architecture and deployment patterns is key to successful use, as is knowing where to go looking when things aren't working.

This talk covers:

* key design concepts within Kafka Connect

* Deployment modes

* Live demo

* Diagnosing and resolving common issues encountered with Kafka Connect.

* Single Message Transforms

* Deployment of Kafka Connect in containers.

📔 Slides: https://rmoff.dev/kafka-connect-zero-to-hero

👾 Code: https://github.com/confluentinc/demo-scene/tree/master/kafka-connect-zero-to-hero

⏱ Time codes:

00:00 What is Kafka Connect?

03:38 Demo streaming data from MySQL into Elasticsearch

11:43 Configuring Kafka Connect

12:33 👉 Connector plugins

13:33 👉 Converters

13:53 👉 Serialisation and Schemas (Avro, Protobuf, JSON Schema)

17:13 👉 Single Message Transforms

19:43 👉 Confluent Hub

19:51 Running Kafka Connect

20:24 👉 Connectors and Tasks

21:29 👉 Workers

21:56 👉 Standalone Worker

22:50 👉 Distributed Worker

23:10 👉 Scaling Kafka Connect

24:42 Kafka Connect on Docker

26:17 Troubleshooting Kafka Connect

27:56 👉 Dynamic Log levels in Kafka Connect

28:48 👉 Error handling and Dead Letter Queues

32:16 Monitoring Kafka Connect

32:59 Recap & Resources

--

☁️ Confluent Cloud ☁️

Confluent Cloud is a managed Apache Kafka and Confluent Platform service. It scales to zero and lets you get started with Apache Kafka at the click of a mouse. You can signup at https://confluent.cloud/signup?utm_source=youtube&utm_medium=rmoff&utm_campaign=ty.community.con.rmoff_youtube_2fUOi9wJPhk&utm_term=rmoff-devx and use code 60DEVADV for $60 towards your bill (small print: https://www.confluent.io/confluent-cloud-promo-disclaimer/)

--

💾Download Confluent Platform: https://www.confluent.io/download/?utm_source=youtube&utm_medium=rmoff&utm_campaign=ty.community.con.rmoff_youtube_RQn3tYvkeh8&utm_term=rmoff-devx

📺 Kafka Connect connector deep-dives: https://www.youtube.com/playlist?list=PL5T99fPsK7ppB_AbZhBhTyKHtHWZLWIJ8

✍️Kafka Connect documentation: https://docs.confluent.io/current/connect/?utm_source=youtube&utm_medium=rmoff&utm_campaign=ty.community.con.rmoff_youtube_RQn3tYvkeh8&utm_term=rmoff-devx

🧩Confluent Hub: https://www.confluent.io/hub/?utm_source=youtube&utm_medium=rmoff&utm_campaign=ty.community.con.rmoff_youtube_RQn3tYvkeh8&utm_term=rmoff-devx