Is it possible to add partitions to an existing topic in Kafka 0.8.2

I have a Kafka cluster running with 2 partitions. I was looking for a way to increase the partition count to 3. However, I don't want to lose existing messages in the topic. I tried stopping Kafka, modifying the server.properties file to increase the number of partitions to 3 and restart Kafka. However, that does not seem to change anything. Using Kafka ConsumerOffsetChecker, I still see it is using only 2 partitions. The Kafka version I am using is 0.8.2.2. In version 0.8.1, there used to be a script called kafka-add-partitions.sh, which I guess might do the trick. However, I don't see any such script in 0.8.2. Is there any way of accomplishing this? I did experiment with creating a whole new topic and for that one it does seem to use 3 partitions as per the change in the server.properties file. However, for existing topics, it doesn't seem to care.


Looks like you can use this script instead:

bin/kafka-topics.sh --zookeeper zk_host:port/chroot --alter --topic my_topic_name 
   --partitions 40 

In the code it looks like they do same thing:

 AdminUtils.createOrUpdateTopicPartitionAssignmentPathInZK(topic, partitionReplicaList, zkClient, true)

kafka-topics.sh executes this piece of code as well as AddPartitionsCommand used by kafka-add-partition script.

However you have to be aware of re-partitioning when using key:

Be aware that one use case for partitions is to semantically partition data, and adding partitions doesn't change the partitioning of existing data so this may disturb consumers if they rely on that partition. That is if data is partitioned by hash(key) % number_of_partitions then this partitioning will potentially be shuffled by adding partitions but Kafka will not attempt to automatically redistribute data in any way.

Adding Partitions to a Topic in Apache Kafka, Can I add a partition to an existing Kafka topic? Apache Kafka provides us with alter command to change Topic behaviour and add/modify configurations. We will be using alter command to add more partitions to an existing Topic. Here is the command to increase the partitions count from 2 to 3 for topic 'my-topic' -./bin/kafka-topics.sh --alter --zookeeper localhost:2181 --topic my-topic


For anyone who wants solution for newer Kafka versions.Please follow this method.

Kafka's entire data retention and transfer policy depends on partitions so be careful about effects of increasing partitions. (Kafka's newer versions display warning regarding this) Try to avoid configuration in which one broker has too many leader partitions.

There is simple 3 stage approach to this.

Step 1: Increase the partitions in topics

./bin/kafka-topics.sh --zookeeper localhost:9092 --alter --topic testKafka_5 --partitions 6

Step 2: Create a partitioning json file for given topic

{ "version":1, "partitions":[ {"topic":"testKafka_5","partition":0,"replicas":[0,1,2]}, {"topic":"testKafka_5","partition":1,"replicas":[2,1,0]}, {"topic":"testKafka_5","partition":2,"replicas":[1,2,0]}, {"topic":"testKafka_5","partition":3,"replicas":[0,1,2]}, {"topic":"testKafka_5","partition":4,"replicas":[2,1,0]}, {"topic":"testKafka_5","partition":5,"replicas":[1,2,0]} ]}

Create file with newer partition and replicas. It's better to expand replicas to different brokers but they should be present within same cluster. Take latency into consideration for distant replicas. Transfer the given file to your Kafka.

Step 3: Reassign partitions and verify

./bin/kafka-reassign-partitions.sh --zookeeper localhost:9092 --reassignment-json-file bin/increase-replication-factor.json  --execute

./bin/kafka-reassign-partitions.sh --zookeeper localhost:9092 --reassignment-json-file bin/increase-replication-factor.json --verify

You can check the effects of your change using --describe command.

Kafka Optimization, How do I change the number of partitions in Kafka? For such topics, it is highly recommended that the partitions are set at the time of creation and avoid resizing later (Reference: Kafka Definitive Guide - Chapter 9 Administering Kafka -> Topic Operations -> Adding Partitions). I've just tried to support the explanation with the above examples but there may be more scenarios. Hope this helps!


In my case the value zk_host:port/chroot for parameter --zookeeper threw the following exception:

ERROR java.lang.IllegalArgumentException: Topic my_topic_name does not exist on ZK path zk_host:port/chroot.

So, I tried the following and it worked:

 bin/kafka-topics.sh --alter --zookeeper zk_host:port --topic my_topic_name --partitions 10

what happens after a broker is down in a cluster?, of a Topic. In order to achieve this, you need to delete and re-create your Topic. When you are starting your kafka broker you can define a bunch of properties in conf/server.properties file. One of the property is auto.create.topics.enable if you set this to true (by default) kafka will automatically create a topic when you send a message to a non existing topic.


If you are using Kafka in Windows try this code for alter or add partition in topic

.\bin\windows\kafka-topics.bat --alter --zookeeper localhost:2181 --topic TopicName --partitions 20

or

.\bin\windows\kafka-topics.bat --alter --zookeeper localhost:2181 --topic TopicName --replica-assignment 0:1:2,0:1:2,0:1:2,2:1:0 --partitions 10

Documentation - Apache Kafka, per Kafka cluster. Going beyond that amount can require additional monitoring and optimization. Utilisation De Kafka ConsumerOffsetChecker, je vois encore, il est à l'aide de seulement 2 partitions. La version Kafka que j'utilise est 0.8.2.2. Dans la version 0.8.1, il y avait un script appelé kafka-add-partitions.sh, qui je pense pourrait faire l'affaire. Cependant, je ne vois pas un tel script dans 0.8.2. Est-il possible de réaliser cela?


Post Deployment, , it will try to get back in sync. Once done, whether it stays a follower or becomes the leader depends if it is the prefered replica. You can change the configuration or partitioning of a topic using the same topic tool. To add partitions you can do > bin/kafka-topics.sh --zookeeper zk_host:port/chroot --alter --topic my_topic_name --partitions 40


yahoo/CMAK: CMAK is a tool for managing Apache Kafka , This tutorial assumes you are starting fresh and have no existing Kafka or ZooKeeper data. Let's create a topic named "test" with a single partition and only one replica: you should now be able to type messages into the producer terminal and see we plan to add tooling support for decommissioning brokers in 0.8.2. kafka-topics --zookeeper localhost:2181 --create --topic test --partitions 3 --replication-factor 1 We have to provide a topic name, a number of partitions in that topic, its replication factor along with the address of Kafka’s zookeeper server.


herokumx/heroku-kafka-manager - Buttons, the instructions to add partitions to an existing Topic in Apache Kafka. Note: While Kafka allows us to add more partitions, it is NOT possible  Here is an example for a kafka cluster with 10 brokers, 100 topics, with each topic having 10 partitions giving 1000 total partitions with JMX enabled : kafka-manager.broker-view-thread-pool-size=30 kafka-manager.broker-view-max-queue-size=3000