Prerequisites

Before you begin, ensure you have the following installed:

• Java: Kafka requires Java to run. You can download it from Oracle's Java downloads.
• Apache Kafka: Download Kafka from the Kafka website and extract the files to a directory on your system.
• .NET Core SDK: Required to write C# producer and consumer code. Download it from Microsoft’s .NET downloads.

Step 1: Start Kafka and ZooKeeper

Kafka relies on ZooKeeper for cluster management. Start ZooKeeper first, then Kafka:

# Start ZooKeeper
bin/zookeeper-server-start.sh config/zookeeper.properties

# Start Kafka
bin/kafka-server-start.sh config/server.properties

These commands will start ZooKeeper on port 2181 and Kafka on port 9092 by default. Ensure both are running before moving on.

Step 2: Create a Kafka Topic

Topics in Kafka organize data streams. Use the following command to create a topic named test-topic:

bin/kafka-topics.sh --create --topic test-topic --bootstrap-server localhost:9092 --partitions 1 --replication-factor 1

This command creates a single-partition topic with a replication factor of 1. To verify, list all topics:

bin/kafka-topics.sh --list --bootstrap-server localhost:9092

Step 3: Implement a C# Kafka Producer

Let’s build a producer in C# that sends messages to test-topic. Start by installing the Confluent.Kafka NuGet package:

dotnet add package Confluent.Kafka

Here’s a simple producer that sends messages to Kafka:

using System;
using System.Threading.Tasks;
using Confluent.Kafka;

class KafkaProducer
{
    public static async Task Main(string[] args)
    {
        var config = new ProducerConfig { BootstrapServers = "localhost:9092" };

        using (var producer = new ProducerBuilder<string, string>(config).Build())
        {
            for (int i = 0; i < 5; i++)
            {
                var message = $"Message {i}";
                await producer.ProduceAsync("test-topic", new Message<string, string> { Value = message });
                Console.WriteLine($"Produced: {message}");
            }

            producer.Flush(TimeSpan.FromSeconds(10));
        }
    }
}

This producer sends five messages to test-topic and displays each produced message. Run the code to confirm that messages are being sent to Kafka.

Step 4: Implement a C# Kafka Consumer

Now let’s create a consumer that reads messages from test-topic in real-time. Use the following code:

using System;
using Confluent.Kafka;

class KafkaConsumer
{
    public static void Main(string[] args)
    {
        var config = new ConsumerConfig
        {
            GroupId = "test-consumer-group",
            BootstrapServers = "localhost:9092",
            AutoOffsetReset = AutoOffsetReset.Earliest
        };

        using (var consumer = new ConsumerBuilder<string, string>(config).Build())
        {
            consumer.Subscribe("test-topic");

            while (true)
            {
                var result = consumer.Consume();
                Console.WriteLine($"Consumed: {result.Message.Value}");
            }
        }
    }
}

This consumer continuously listens to test-topic and displays each message. Run the producer code to generate messages, then run the consumer to view the consumed messages.

Step 5: Testing the Producer and Consumer

With the producer and consumer in place, it’s time to test your setup. Run the KafkaProducer program to send messages to test-topic, then start the KafkaConsumer program to retrieve and print those messages. You should see each message produced by the producer displayed by the consumer.

Conclusion

Congratulations! You’ve built your first Kafka project, complete with a producer and consumer. This tutorial covered the basics of creating a topic, producing messages, and consuming them in real time using C#. By following these steps, you now have a foundational Kafka setup and a working understanding of its core components. As you become more comfortable with Kafka, explore additional features like stream processing, advanced configurations, and multi-topic setups to expand your capabilities.