How to Configure and Test a Single-Node Apache Kafka Cluster

In the previous article we have downloaded the Apache Kafka and also we have extracted the Apache Kafka downloaded file now it is the time to test the Apache Kafka. By default, Kafka stores data in the temporary /tmp directory, which means your data could vanish after a reboot.

In this article, we will configure persistent storage paths for both Zookeeper and Kafka Broker, and then demonstrate a full "Hello World" cycle: creating a topic, producing messages, and consuming them.

Step 1: Configure Zookeeper

Kafka uses Zookeeper to coordinate the cluster. We need to edit its configuration to ensure it stores data in a permanent directory rather than a temporary one.

  • Navigate to the config directory: Copy the below command and paste in the terminal to change the directory.
          cd kafka/config
  • Open the properties file: Copy the below and paste in the terminal to open the file.
    nano zookeeper.properties

  • Modify the dataDir: Find the line starting with dataDir=/tmp/zookeeper and change it to your desired path (e.g., inside your home directory).

       Change:

       dataDir=/tmp/zookeeper

   To:

        dataDir=/home/kafkausernew/zookeeper-kafka

       (Note: Ensure the directory /home/ubuntu/zookeeper-kafka exists or that the user has permissions to create it.)

  • Save and Exit: Press CTRL + X, type Y, and hit ENTER.

Step 2: Configure the Kafka Broker

Next, we will configure the Kafka server (Broker) to store its logs (the actual data messages) in a permanent location.

  • Open the server properties file: In the same terminal write the below command and do the configuration inside the server.properties file:

     nano server.properties 

  • Modify the log.dirs: Find the line for log.dirs and update it to a permanent path.
           Change:

               log.dirs=/tmp/kafka-logs

           To:
              log.dirs=/home/ubuntu/kafka-logs
  • Save and Exit: Press CTRL + X, type Y, and hit ENTER.

Step 3: Start the Services

Now that our configuration is secure, let's start the cluster.

         1. Start Zookeeper

           Run the following command in your terminal to run the zookeeper:
                    bin/zookeeper-server-start.sh config/zookeeper.properties

    2. Start Kafka Broker

           Open a new terminal window and type the below command to run the Kafka server:
                        bin/kafka-server-start.sh config/server.properties

Step 4: Test the Installation

Now both Zookeeper and the Kafka Broker is running, we can now verify the system by sending and receiving data.

        1. Create a Topic

       A "Topic" is a category or feed name to which records are published. Let's create a topic named test.
  • In the new terminal type the below command to create the topic
      bin/kafka-topics.sh --create --topic test --bootstrap-server localhost:9092
Output:

      2. Produce Messages (The Publisher)

  • Now, let's become a "Producer" and write data into the test topic.
  • Now in another terminal follow the below command to run the producer:
               bin/kafka-console-producer.sh --topic test --bootstrap-server localhost:9092
  • You will see a > prompt. Type a few messages and press Enter after each one:            

      3. Consume Messages (The Subscriber)      

  • Finally, let's verify that the messages were stored. We will start a "Consumer" to read from the test topic.
  • Open a new terminal window and run the consumer script to read messages from the topic:
  bin/kafka-console-consumer.sh --topic test --bootstrap-server localhost:9092 --from-beginning

Output: You should immediately see the messages you typed in the producer terminal appear here:

Conclusion

Now we have successfully configured a persistent Kafka single-node cluster. You have also verified the pipeline by acting as both a Producer (writing data) and a Consumer (reading data).








Comments

Post a Comment

Popular posts from this blog

Virtual Private Network - VPN

Image
VPN  stands for   Virtual Private Network. It is a technology that creates a safe and encrypted tunnel over a less secure network, such as the public internet. A VPN allows you to securely access a private network and share data remotely. In simple terms, a Virtual Private Network gives you online privacy and anonymity by creating a private network from a public internet connection. It hides your Internet Protocol (IP) address, making your online actions virtually untraceable.
Read more

Windows Registry Forensics: Detecting Malware Persistence with Process Monitor

Image
In the world of cybercrime investigation, the Windows Registry isn't just a database—it’s a digital diary . The Registry keeps track of nearly everything that occurs on a computer, including user behavior and system configurations . Two methods become essential for an investigator when a system is suspected of being infected: Boot Time Logging and Registry Analysis . In this guide, we will explore how to use Process Monitor (ProcMon) to look inside a Windows system and find malicious activity that may be hidden.
Read more

Mastering Incident Response: Complete Guide to CrowdResponse Forensic Tool

Image
When a security incident occurs, time is critical. Responders must collect volatile evidence —running processes, network connections, registry data—before it is lost or altered. However, installing complex forensic software on a compromised system can be slow and may contaminate evidence. CrowdResponse  offers a practical solution. Developed by CrowdStrike and available as a free,  CrowdResponse tool is  a  lightweight, portable tool allows incident responders to gather key system artifacts quickly without installation. It runs directly from a USB drive or local directory, making it ideal for live triage investigations on Windows systems. In this article, we will learn how to deploy CrowdResponse on Windows systems, master all 16 modules with practical examples, and analyze collected data to identify security threats.
Read more