This project is a k6 extension that can be used to load test Kafka, using a producer. Per each connection to Kafka, many messages can be sent. These messages are an array of objects containing a key and a value. There is also a consumer for testing purposes, i.e. to make sure you send the correct data to Kafka. The consumer is not meant to be used for testing Kafka under load. The extension supports producing and consuming messages in Avro format, given a schema for key and/or value.
The real purpose of this extension is not only to test Apache Kafka, but also the system you've designed that uses Apache Kafka. So, you can test your consumers, and hence your system, by auto-generating messages and sending them to your system via Apache Kafka.
If you want to learn more about the extension, visit How to Load Test Your Kafka Producers and Consumers using k6 article on the k6 blog.
To build a
k6 binary with this extension, first ensure you have the prerequisites:
Then, install xk6 and build your custom k6 binary with the Kafka extension:
$ go install go.k6.io/xk6/cmd/xk6@latest
Build the binary:
$ xk6 build --with github.com/mostafa/xk6-kafka@latest
Note: you can always use the latest version of k6 to build the extension, but the earliest version of k6 that supports extensions via xk6 is v0.32.0.
First, you need to have your Kafka development environment setup. I recommend you to use Lenses.io fast-data-dev Docker image, which is a complete Kafka setup for development that includes: Kafka, Zookeeper, Schema Registry, Kafka-Connect, Landoop Tools, 20+ connectors. It is fairly easy to setup, if you have Docker installed. Just make sure to monitor Docker logs to have a working setup, before attempting to test. Initial setup, leader election and test data ingestion takes time.
Run the Kafka environment and expose the container ports:
sudo docker run -d --rm --name lenseio -p 2181:2181 -p 3030:3030 \ -p 8081-8083:8081-8083 -p 9581-9585:9581-9585 -p 9092:9092 \ -e ADV_HOST=127.0.0.1 lensesio/fast-data-dev
After running the command, visit localhost:3030 to get into the fast-data-dev environment.
You can run the command to see the container logs:
sudo docker logs -f -t lensesio
If you have errors running the Kafka development environment, refer to the fast-data-dev documentation.
The following k6 test script is used to test this extension and Apache Kafka in turn. The script is available as
test_<format>.js with more code and commented sections. The scripts have 4 parts:
checkfrom k6 and the
consumefrom the extension using the
k6/x/kafkaextension loading convention.
writerfunction is used to open a connection to the bootstrap servers. The first argument is an array of strings that signifies the bootstrap server addresses and the second is the topic you want to write to. You can reuse this writer object to produce as many messages as you want. This object is created in init code and is reused in the exported default function.
producefunction is used to send a list of messages to Kafka. The first argument is the
producerobject, the second is the list of messages (with key and value). The third and the fourth arguments are the key schema and value schema in Avro format, if Avro format is used. If the schema are not passed to the function for either the key or the value, the values are treated as normal strings. Use an empty string,
""if either of the schema is Avro and the other is going to be a string. The produce function returns an
errorif it fails. The check is optional, but
producefunction successfully sent the message.
producer.close()function closes the
readerfunction is used to open a connection to the bootstrap servers. The first argument is an array of strings that signifies the bootstrap server addresses and the second is the topic you want to reader from. This object is created in init code and is reused in the exported default function.
consumefunction is used to read a list of messages from Kafka. The first argument is the
consumerobject, the second is the number of messages to read in one go. The third and the fourth arguments are the key schema and value schema in Avro format, if Avro format is used. If the schema are not passed to the function for either the key or the value, the values are treated as normal strings. Use an empty string,
""if either of the schema is Avro and the other is going to be a string. The consume function returns an empty array if it fails. The check is optional, but it checks to see if the length of the message array is exactly 10.
consumer.close()function closes the
You can run k6 with the Kafka extension using the following command:
$ ./k6 run --vus 50 --duration 60s scripts/test_json.js
And here's the test result output:
/\ |‾‾| /‾‾/ /‾‾/ /\ / \ | |/ / / / / \/ \ | ( / ‾‾\ / \ | |\ \ | (‾) | / __________ \ |__| \__\ \_____/ .io execution: local script: scripts/test_json.js output: - scenarios: (100.00%) 1 scenario, 50 max VUs, 1m30s max duration (incl. graceful stop): * default: 50 looping VUs for 1m0s (gracefulStop: 30s) running (1m00.4s), 00/50 VUs, 6554 complete and 0 interrupted iterations default ✓ [======================================] 50 VUs 1m0s ✓ is sent ✓ 10 messages returned checks.........................: 100.00% ✓ 661954 ✗ 0 data_received..................: 0 B 0 B/s data_sent......................: 0 B 0 B/s iteration_duration.............: avg=459.31ms min=188.19ms med=456.26ms max=733.67ms p(90)=543.22ms p(95)=572.76ms iterations.....................: 6554 108.563093/s kafka.reader.dial.count........: 6554 108.563093/s kafka.reader.error.count.......: 0 0/s kafka.reader.fetches.count.....: 6554 108.563093/s kafka.reader.message.bytes.....: 6.4 MB 106 kB/s kafka.reader.message.count.....: 77825 1289.124612/s kafka.reader.rebalance.count...: 0 0/s kafka.reader.timeouts.count....: 0 0/s kafka.writer.dial.count........: 6554 108.563093/s kafka.writer.error.count.......: 0 0/s kafka.writer.message.bytes.....: 54 MB 890 kB/s kafka.writer.message.count.....: 655400 10856.309293/s kafka.writer.rebalance.count...: 6554 108.563093/s kafka.writer.write.count.......: 655400 10856.309293/s vus............................: 50 min=50 max=50 vus_max........................: 50 min=50 max=50
To avoid getting the following error while running the test:
Failed to write message:  Leader Not Available: the cluster is in the middle of a leadership election and there is currently no leader for this partition and hence it is unavailable for writes
You can now use
createTopic function to create topics in Kafka. The
scripts/test_topics.js script shows how to list topics on all Kakfa partitions and also how to create a topic.
You always have the option to create it using
$ docker exec -it lensesio bash (inside container)$ kafka-topics --create --topic xk6_kafka_avro_topic --bootstrap-server localhost:9092 (inside container)$ kafka-topics --create --topic xk6_kafka_json_topic --bootstrap-server localhost:9092
If you want to test SASL authentication, have a look at this commmit message, where I describe how to run a test environment.
This is a proof of concept, isn't supported by the k6 team, and may break in the future. USE AT YOUR OWN RISK!
This work is licensed under the Apache License 2.0.
Build REST APIs with Neo4j and Flask, as quickly as possible!
A Visual Studio Code extension for auto typing.
Benchmark of Kafka libraries for Go
Affogato is a multipurpose library for Android, Jetpack Compose and Kotlin.
This Library a Audio Validator For Django With d MIME type checking the magic numbers signature And Extension Use FileType Library
Goja debugger's CLI frontend
A fault-tolerant, protocol-agnostic, dead-simple, reliable, interoprable and secure RPC framework
Performance Testing gRPC Services