Implementing a Kafka Producer in C++

Apache Kafka, a distributed streaming platform, is a valuable tool for handling data streams. In this guide, we’ll walk through the steps to implement a Kafka producer in C++. This will enable you to efficiently send data to a Kafka topic and integrate it into your data processing pipelines.

blocks

Image by Benjamin Zocholl from Pixabay

To illustrate the process, the project kafka-producer contains the source code.

Setting Up Your Development Environment

Before you start coding, ensure you have the necessary tools and libraries installed, including a C++ compiler and the librdkafka C/C++ library, which is a popular choice for Kafka integration in C++. You can install it via package managers or compile it from source. We will also use CMake, GTest, and GMock installed on an Ubuntu 22.04 LTS System to provide a rock-solid foundation for further development of your Producer.

  1. Install the required libraries to your Ubuntu machine: 
    sudo apt-get update && sudo apt-get install -y \
   git \
   build-essential \
   cmake \
   clang \
   libcurl4-openssl-dev \
   liblz4-dev \
   zlib1g-dev \
   libssl-dev \
   libsasl2-dev
  2. Download the latest version of librdkafka 
    LIBRDKAFKA_VERSION="v2.2.0" && git clone --branch $LIBRDKAFKA_VERSION \
https://github.com/confluentinc/librdkafka.git librdkafka
  3. Compile the library for cmake and installed on the machine 
    cd librdkafka
cmake -H. -B_cmake_build 
cmake --build build -j 6
cmake --build _cmake_build --target install

The Docker File can be used to help reproduce and have the compiled library.

Creating a Kafka Producer

To build a Kafka producer in C++, you’ll need to use the librdkafka library.

Here are the basic steps of how to create a producer:

  1. Create a configuration to connect to a broker

    std::unique_ptr<RdKafka::Conf> conf(RdKafka::Conf::create(RdKafka::Conf::CONF_GLOBAL));
conf->set("bootstrap.servers", brokersList, error_str);

  2. Create a Kafka producer

    std::unique_ptr<RdKafka::Producer> producer(RdKafka::Producer::create(conf.get(), error_str));

  3. Send a message

    RdKafka::ErrorCode producerError = kafkaProducer->produce(
                 topic,
                 RdKafka::Topic::PARTITION_UA,
                 RdKafka::Producer::RK_MSG_COPY,
                 const_cast<char *>(message.c_str()), message.size(),
                 /* Key */
                 NULL, 0,
                 /* Timestamp (defaults to current time) */
                 0,
                 /* Message headers, if any */
                 NULL,
                 /* Per-message opaque value passed to
                  * delivery report */
                 NULL
         );

The complete code for the producer can be found here and the test here. This code sets up a Kafka producer, sends a message to a specified topic, and handles any potential errors.

Conclusion

By following this guide, you’ve learned how to implement a Kafka producer in C++ using the librdkafka library. This producer can be a valuable component in your data processing pipelines, enabling you to efficiently send data to Kafka topics and integrate it into your real-time data processing workflows.