I am writing this based on my experience of using Kafka confluent with Node JS, let’s first check on Kafka

What is Apache Kafka?

Apache Kafka is a distributed data store optimized for ingesting and processing streaming data in real-time. Streaming data is data that is continuously generated by thousands of data sources, which typically send the data records in simultaneously. A streaming platform needs to handle this constant influx of data, and process the data sequentially and incrementally.

Kafka provides three main functions to its users:

• Publish and subscribe to streams of records
• Effectively store streams of records in the order in which records were generated
• Process streams of records in real-time

Kafka is primarily used to build real-time streaming data pipelines and applications that adapt to the data streams. It combines messaging, storage, and stream processing to allow storage and analysis of both historical and real-time data

Kafka messages are persisted on the disk and replicated within the cluster to prevent data loss. Kafka is built on top of the ZooKeeper synchronization service. It integrates very well with Apache Storm and Spark for real-time streaming data analysis

In this blog, we are talking about how to connect and build your service with Kafka where the Kafka cluster is already there

it's a simple picture we have Kafka Platform ready from https://confluent.cloud/ its Kafka platform provider from where we can buy this service and can start using it, it's like managed solutions provided by AWS

Now we can send or stream messages to Kafka where consumers can consume and react on that message, so what do we need for doing this, some library and Kafka connection details from https://confluent.cloud/

Let's check different options

I started with node-rdkafka and later I moved to kafkajs but why ??

• Node.js version compatibility can cause problems with node-rdkafka.
• Use the OS installation of librdkafka or build from source.
• If you use node-rdkafka, you are bound to encounter compatibility issues as you upgrade the library or versions of Node.js. I recommend you use a system installation of librdkafka and the BUILD_LIBRDKAFKA=0 flag to prevent the recompilation of the library on npm install. Configuring Kafka can be complicated — https://rclayton.silvrback.com/thoughts-on-node-rdkafka-development

UnhandledPromiseRejectionWarning: Error: Unsupported value "sasl_ssl" for configuration property "security.protocol": OpenSSL not available at build time
at Producer.Client (/Users/node_modules/node-rdkafka/lib/client.js:54:18)
at new Producer (/Users/node_modules/node-rdkafka/lib/producer.js:75:10)

• user compatible node version with node-rdkafka
• possible error `UnhandledPromiseRejectionWarning: Error: Unsupported value “sasl_ssl” for configuration property “security.protocol”: OpenSSL not available at build time` fix is link OpenSSL properly
• You can see if you can fix by linking open SSL properly

brew link openssl --force 
export LDFLAGS="-L/usr/local/opt/[email protected]/lib" 
export CPPFLAGS="-I/usr/local/opt/[email protected]/include" 
echo 'export PATH="/usr/local/opt/[email protected]/bin:$PATH"' >> ~/.zshrc 
npm rebuild node-rdkafka

So the conclusion is I am not a big fan of node-rdkafka

Here is the better solution Kafkajs

kafka js is a native library without any node js binding so there will be no compatibility issues and no runtime errors

So with kafka js, lots of problems with integration are no more for the developers

References