Contact Us

Contact Us



Pleas confirm by checkbox


Technical

Kafka Benchmarking

Author_img
By Gaurav Harsola November 30, 2016

Kafka Introduction

Before understand the Kafka bench-marking, let me give a quick brief of what Kafka is and a few details about how it works. Kafka is a distributed messaging system originally built at LinkedIn and now part of Apache Software Foundation and used by variety of companies.

The general setup is quite simple. Producers send records to the cluster which holds on to these records and hands them out to consumers:

The key abstraction in Kafka is the topic. Producers publish their records to a topic, and consumers subscribe to one or more topics. A Kafka topic is just a shared write-ahead log. Producers append records to these logs and consumers subscribe to changes. Each record is a key/value pair.

Here is a simple example of a single producer and consumer reading and writing from a two-partition topic.

Kafka Benchmarking

Scenario 1:

All of these benchmarks were run on AWS EC2 instances with the following specifications:

  • Kafka cluster is running on three Machines; the spec of each Machine is Instance Type:-c3.4xlarge, 16 VCPUs, 30 GB RAM, Disk :- 2 * 160 GB (SSD’s) ,RAID:- 0 (Stripping) and XFS Filesystem
  • Zookeeper running on three mahcines; the spec of each machine is of t2.micro

These benchmarks were run for the following cases:

  • Single producer (single thread) and single consumer producing asynchronousyly on a topic with replication factor of 3 and 1 partitions.
  1. 9 producer (3 producer with three thread so total 9 (3*3) ) and 3 consumer producing asynchronously on a topic with replication factor of 3x and 1 partitions.

In this context, a producer running in async mode would mean that the number of acknowledgements the producer requires the leader to have received from its followers before considering a request complete is 0.

Case 1: Single producer (single thread) and Single Consumer producing asynchronously on a topic with replication factor of 3 and 1 partitions.

We have performed the test for various records(2 ,5,10 and 15 lakh) with different record size (10KB,5KB,2KB) by varying throughput (1000 ,5000 and 10000 )

Command:

Producer test performance command :

./bin/kafka-run-class.sh org.apache.kafka.tools.ProducerPerformance –topic <topic-name> –num-records 1000000 –record-size <record-size> –producer-props bootstrap.servers=<broker-list> –throughput <value>

Consumer test performance command:

bin/kafka-consumer-perf-test.sh –zookeeper <zk-list> –threads 1 –topic <topic-name> –messages 1000000

Result:

Conclusion:

1) With decrease in record size for a message the average latency decreases

2) More Throughput more Latency

Latency

3) Throughout and Record -Size are inversely proportional.With more Record-size we have less throughput.

Case 2:  9 producer (3 producer with three thread so total 9 (3*3) ) and 3 Consumer producing asynchronously on a topic with replication factor of 3x and 1 partitions.

Producer

Consumer

Conclusion:

By Increasing the number of Producer(compared to first case) the Latency of the producer also increases .

Scenario 2:

All of these benchmarks were run on AWS EC2 instances with the following specifications:

  • Kafka cluster is running on three Machines; the spec of each Machine is c3.4xlarge, 16 VCPUs, 2 * 160 GB (SSD’s), EBS ST1 disk type, EXT4 filesystem
  • Zookeeper running on three machines; the spec of each machine is of t2.micro

These benchmarks were run for the following cases:

1.Single producer (single thread) and single consumer producing asynchronously on a topic with replication factor of 3 and 1 partitions.

2. 9 producer (3 producer with three thread so total 9 (3*3) ) and 3 consumer producing asynchronously on a topic with replication factor of 3x and 1 partitions.

Case 1: Single producer (single thread) and Single Consumer producing asynchronously on a topic with replication factor of 3 and 1 partitions.

Result:

If we compare above result with Scenario 1 case 1 result ,we found that the avergae latency with EBS ST1 disk is very low.

 

Case 2:  9 producer (3 producer with three thread so total 9 (3*3) ) and 3 consumer producing asynchronously on a topic with replication factor of 3x and 1 partitions.

Producer

Consumer

Conclusion:

With increase in producer count (compared to above Scenario 2 case 1) the Latency of the producer also increases .

Related posts
Apache Spark Standalone Setup On Linux/macOS
Technical

Apache Spark Standalone Setup On Linux/macOS

By shekhar.wagh October 20, 2021
Apache Flink Standalone Setup on Linux/macOS
Technical

Apache Flink Standalone Setup on Linux/macOS

By shekhar.wagh October 13, 2021
Identity, Authentication, And Access Management Using Azure Active Directory  
Technical

Identity, Authentication, And Access Management Using Azure Active Directory  

By shekhar.wagh September 22, 2021
Things to Know Before You Select A Crypto Wallet
Blockchain

Things to Know Before You Select A Crypto Wallet

By shekhar.wagh September 20, 2021
Solve 3 Most Irritating Outlook Email Rendering Issues.
Technical

Solve 3 Most Irritating Outlook Email Rendering Issues.

By shekhar.wagh September 15, 2021
Intuit Wasabi – A Scalable A/B Testing Solution
Technical

Intuit Wasabi – A Scalable A/B Testing Solution

By shekhar.wagh September 01, 2021
How To Pick The Right Data Analytics Strategy For Serverless Systems?
Big Data

How To Pick The Right Data Analytics Strategy For Serverless Systems?

By shekhar.wagh August 25, 2021
Change Notifications and Named Options using Options pattern in .NET Core
Technical

Change Notifications and Named Options using Options pattern in .NET Core

By shekhar.wagh August 18, 2021
Create Strongly Typed Configurations in .NET Core
Technical

Create Strongly Typed Configurations in .NET Core

By shekhar.wagh August 13, 2021
Partitioning Database - A Divide and Rule Strategy
Technical

Partitioning Database - A Divide and Rule Strategy

By shekhar.wagh June 02, 2021

Stay updated

Get the latest creative news from Fubiz about art, design and pop-culture.