Step 1: Installing Kafka and Prerequisites

Before installing Kafka, make sure you have Java installed on your system. Kafka requires Java 8 or later, and you can check your Java version by running:

java -version

If Java is not installed, download it from the official Oracle site or install it via your package manager.

Download Kafka

Download the latest version of Kafka from the official Apache Kafka website. After downloading, extract the contents of the compressed file to a preferred directory:

tar -xzf kafka_2.13-2.8.0.tgz
cd kafka_2.13-2.8.0

Start ZooKeeper

Kafka relies on ZooKeeper for managing metadata, so you’ll need to start ZooKeeper first. Run the following command from the Kafka installation directory:

bin/zookeeper-server-start.sh config/zookeeper.properties

ZooKeeper will start running on its default port, 2181.

Start Kafka Broker

With ZooKeeper running, you can now start a Kafka broker:

bin/kafka-server-start.sh config/server.properties

Kafka will start running on port 9092, the default port. Now, your Kafka setup is ready to handle topics, producers, and consumers.

Step 2: Creating a Topic

Topics are fundamental to Kafka as they organize and store data. To create a topic, use the following command:

bin/kafka-topics.sh --create --topic test-topic --bootstrap-server localhost:9092 --partitions 1 --replication-factor 1

This creates a topic named test-topic with one partition and a replication factor of 1. You can verify its creation with:

bin/kafka-topics.sh --list --bootstrap-server localhost:9092

Step 3: Writing a Simple C# Producer

In this example, we’ll create a basic C# producer using the Confluent.Kafka library, which provides an easy interface for working with Kafka in C#. Install the Confluent.Kafka NuGet package in your project:

Install-Package Confluent.Kafka

Once installed, use the following code to create a simple Kafka producer:

using System;
using System.Threading.Tasks;
using Confluent.Kafka;

class KafkaProducer
{
    public static async Task Main(string[] args)
    {
        var config = new ProducerConfig { BootstrapServers = "localhost:9092" };

        using (var producer = new ProducerBuilder<string, string>(config).Build())
        {
            for (int i = 0; i < 10; i++)
            {
                var value = $"Message {i}";
                await producer.ProduceAsync("test-topic", new Message<string, string> { Key = null, Value = value });
                Console.WriteLine($"Produced: {value}");
            }

            producer.Flush(TimeSpan.FromSeconds(10));
        }
    }
}

This code snippet creates a producer that sends ten messages to the test-topic. Each message is logged to the console after production.

Step 4: Writing a Simple C# Consumer

Now that we’ve sent data to Kafka, let’s retrieve it using a Kafka consumer. Here’s a basic C# consumer that listens to test-topic and prints incoming messages:

using System;
using Confluent.Kafka;

class KafkaConsumer
{
    public static void Main(string[] args)
    {
        var config = new ConsumerConfig
        {
            GroupId = "test-consumer-group",
            BootstrapServers = "localhost:9092",
            AutoOffsetReset = AutoOffsetReset.Earliest
        };

        using (var consumer = new ConsumerBuilder<string, string>(config).Build())
        {
            consumer.Subscribe("test-topic");

            while (true)
            {
                var result = consumer.Consume();
                Console.WriteLine($"Consumed: {result.Message.Value}");
            }
        }
    }
}

This consumer continuously listens to test-topic and prints each message it receives. You can test it by running the producer code to send messages while this consumer is running.

Step 5: Testing Your Kafka Setup

With both the producer and consumer set up, you’re ready to test your Kafka installation. Run the KafkaProducer program to produce messages to test-topic and start the KafkaConsumer program to consume them. You should see messages produced by KafkaProducer being printed by KafkaConsumer, confirming that your setup works.

Conclusion

Setting up Kafka might seem complex at first, but once you have the essentials running, it’s easy to start experimenting with producers and consumers. With this setup guide, you now have a basic Kafka environment and a simple C# example to begin exploring real-time data streaming. Kafka’s power lies in its scalability and flexibility, so continue experimenting to unlock its full potential in your applications.