How to decrease number partitions Kafka topic?

kafka delete topic
kafka-reassign-partitions
kafka list topics
kafka increase partitions
kafka manual partition assignment
kafka partition example
kafka partition key
kafka default number of partitions

I created a topic with 4 partitions on Kafka. (set default number.partition=4). Now I want to change number partition of this topic to 3. I've tried running

./bin/kafka-topics.sh --alter --zookeeper localhost:2181 --topic my-topic --partitions 3

but there is no change. It still has 4 partitions. Anyone know about this?

Apache Kafka doesn't support decreasing the partition number. You should see the topic as a whole and the partitions are a way for scaling out improving performance. So all data sent to topic flow to all partitions and removing one of them means data loss.

Solved: kafka Topic, How do I change the number of partitions in Kafka? Reducing the number of partitions is not currently supported. Instead, create a new a topic with a lower number of partitions and copy over existing data. Metadata about partitions are stored in ZooKeeper in the form of znodes. Having a large number of partitions has effects on ZooKeeper and on client resources:

You can't just delete a partition because that would lead to data loss and also the remaining data's keys would not be distributed correctly so new messages would not get directed to the same partitions as old existing messages with the same key.

For the above reasons Kafka does not support decreasing partition counts on an existing topic.

What you can do is to create a new topic with 3 partitions and then write an small program (or use an existing replication tool) to copy the data from the old 4 partition topic to the new 3 partition topic. That way you will be running everything through the same partitioner and all your keyed messages will end up in the right partition. Once you are satisfied the data is all copied then delete the original 4 partition topic.

If you must retain the same topic name as the original topic then just create a new topic with the original name, copy the data back from the repartitioned topic, and delete that temporary repartitioning topic.

Adding Partitions to a Topic in Apache Kafka, of a Topic. In order to achieve this, you need to delete and re-create your Topic. By default, whenever a consumer enters or leaves a consumer group, the brokers rebalance the partitions across consumers, meaning Kafka handles load balancing with respect to the number of partitions per application instance for you. This is great—it’s a major feature of Kafka.

Decreasing partition number is not supported.

Kafka Optimization, cluster. Going beyond that amount can require additional monitoring and optimization. # Creates a topic with name 'demo-topic' with 2 partitions and 1 replication factor ./bin/kafka-topics.sh --create --zookeeper localhost:2181 --topic demo-topic --partitions 2 --replication-factor 1. You can verify replicatin factor by using --describe option of kafka-topics.sh as follows -

You can use create standalone java program to achieve the same , i.e increase and decrease the partition and replication using AdminUtils.

import org.I0Itec.zkclient.ZkClient;

import kafka.admin.AdminUtils;

import kafka.utils.ZKStringSerializer$;

import kafka.utils.ZkUtils;

import scala.collection.Seq;

import scala.collection.Map;

public PartitionCreator(String zkhost, String topicName, int partitions, int replifactor) {
    ZkClient zkClient = new ZkClient(zkhost, 30000, 30000, ZKStringSerializer$.MODULE$);
    zkUtils = ZkUtils.apply(zkClient, false);

    this.topicName = topicName;
    this.partitions = partitions;
    this.replifactor = replifactor;
}

public void createPartion() {

    AdminUtils.createTopic(zkUtils, topicName, partitions, replifactor, new Properties());
    System.out.println("created/updated topic..");
}

Note: createTopic() internally updates the topic if topic not available.

kafka get partition count for a topic, if count is 16. You can change count depending on your requirement. Note: While Kafka allows us to add more partitions, it is NOT possible to decrease number of partitions of a Topic. In order to achieve this, you need to delete and re-create your Topic. In order to achieve this, you need to delete and re-create your Topic.

Kafka in a Nutshell, Learn how to determine the number of partitions each of your Kafka topics Instead, create a new a topic with a lower number of partitions and copy over  However, I don't want to lose existing messages in the topic. I tried stopping Kafka, modifying the server.properties file to increase the number of partitions to 3 and restart Kafka. However, that does not seem to change anything. Using Kafka ConsumerOffsetChecker, I still see it is using only 2 partitions. The Kafka version I am using is 0.8.2.2.

Choosing the number of partitions for a topic, More Partitions Lead to Higher Throughput. The first thing to understand is that a topic partition is the unit of parallelism in Kafka. On both the  The Kafka producer api does not allow you to create custom partition, if you try to produce some data to a topic which does not exists it will first create the topic if the auto.create.topics.enable property in the BrokerConfig is set to TRUE and start publishing data on the same but the number of partitions created for this topic will based on the num.partitions parameter defined in the

How to choose the number of topics/partitions in a Kafka cluster , Increase the number of partition from 1 to 3. Just to have more partitions to “play” with. kafka-topics --topic my-topic --alter --partitions 3 --  I got the answer for this question : Number of partitions for a topic can only be increased, never decreased . Reason for this as if we decrease the partition it will be data loss . We can delete current topic and recreate new one with required partition to achieve this .

Comments
  • Another option would be to increase from 4 partitions to 6 partitions and then your 3 consumers would just get 2 partitions each, and still be balanced. You would however lose message ordering across partitions.
  • Thanks your guide, because i can't accept more than one answer so I just accept for the first. But I highly appreciate you.