Inject some data

Let's inject data into our cluster by using Apache Kafka Producer & Consumer.

Producing messages

Using the tool kafka-producer-perf-test, it's possible to inject messages into the cluster.

First of all, open a new terminal and connect to the Kafka broker 1:

docker-compose -f kafka-cluster.yml exec kafka-1 bash

Create the topic my-topic with 6 partitions, and a replication factor set to 3.

unset KAFKA_JMX_OPTS
unset KAFKA_OPTS
kafka-topics --bootstrap-server kafka-1:9092 --create --replication-factor 3 --partitions 6 --topic my-topic

Then, use this commands to inject messages:

kafka-producer-perf-test --topic my-topic --num-records 10800 --record-size 100 --throughput 1 --producer-props bootstrap.servers=kafka-1:9092

Here, this command sent 10800 messages with approximately throughout of one message per second.

TIP

You can use this tool to verify the producer performance. You can specify --producer.config or --producer-props to play with the producer configuration.

Consuming messages

Open a new terminal and connect to the Kafka broker 1:

docker-compose -f kafka-cluster.yml exec kafka-1 bash

Create a console consumer using this command:

unset KAFKA_JMX_OPTS
unset KAFKA_OPTS
kafka-console-consumer --topic my-topic --bootstrap-server kafka-1:9092 --from-beginning

You should see previously produced messages in the console.

TIP

The script kafka-consumer-perf-test can be used to verify consumer performance.