How can I delete Topics using confluent-kafka-python

confluent-kafka-python producer json
confluent-kafka python example
confluent-kafka-python create topic
kafka-python vs confluent-kafka-python
confluent-kafka-python ssl
kafka delete --topic
kafka-python zookeeper
confluent-kafka consumer

I am using Kafka to make multiple micro services communicate with each other. Services are written in Python, and I use Confluent library to handle Kafka. At some point, I know that some Topics are just 'over', so that I could clean them automatically.

Is there a way to delete "Topics" thanks to Confluent library ? I cannot find any documentation on this...

Thanks


You can use the confluent Admin Api's to delete a topic

Example

Takes an AdminClient instance and a list of topics

def example_delete_topics(a, topics):
    """ delete topics """

    # Call delete_topics to asynchronously delete topics, a future is returned.
    # By default this operation on the broker returns immediately while
    # topics are deleted in the background. But here we give it some time (30s)
    # to propagate in the cluster before returning.
    #
    # Returns a dict of <topic,future>.
    fs = a.delete_topics(topics, operation_timeout=30)

    # Wait for operation to finish.
    for topic, f in fs.items():
        try:
            f.result()  # The result itself is None
            print("Topic {} deleted".format(topic))
        except Exception as e:
            print("Failed to delete topic {}: {}".format(topic, e))

confluentinc/confluent-kafka-python, Deleting topic before creating causes race condition #541. Closed. ghost opened this issue on Feb 13, 2019 · 3 comments. Closed  For example, after upgrading Control Center, topics prefixed with previous versions can be safely deleted. Control Center creates internal topics with the current version in use. Select a cluster from the navigation bar. Click the Topics cluster submenu. The All topics page appears. Click the Hide internal topics toggle to off to unhide


To answer your question literally : no, AFAIK you can't delete topics using the client library.

This is possible using the AdminClient API

To address some of your follow up questions, I would suggest there is no point deleting these topics. If you're running a sandbox/prototype, then maybe you end up with lots of topics you want to clear down from experimenting. But once you move these microservices into a Production environment, each topic will serve a purpose, and perhaps it's finished processing the current set of data, but what about the next? Unless you're doing something funky with your topic naming that ties it to some attributes in the data, then topics will be re-used.

The confluent_kafka API, Returns: List of topic+partitions with offset and possibly error set. Return type: Admin¶. Kafka Admin client: create, view, alter, delete topics and resources. topic (str) – If specified, only request info about this topic, else return for all topics in cluster. Warning: If auto.create.topics.enable is set to true on the broker and an unknown topic is specified it will be created.


I use this simple Python function:

def delete_kafka_topic(topic_name):
    call(["/usr/bin/kafka-topics", "--zookeeper", "zookeeper-1:2181", "--delete", "--topic", topic_name])

My team uses this in automated tests, where we want to be able to rerun tests, verify results, and not see results from previous test attempts.

Welcome to Confluent's Apache Kafka Python , List of topic+partitions with offset and possibly error set. Return type: list(​TopicPartition). Raises: KafkaException. get_watermark_offsets ()  topic (str) – If specified, only request info about this topic, else return for all topics in cluster. Warning: If auto.create.topics.enable is set to true on the broker and an unknown topic is specified it will be created. timeout (float) – Maximum response time before timing out, or -1 for infinite timeout.


KafkaAdminClient, Use of this class requires a minimum broker version >= 0.10.0.0. Default: 'kafka​-python-{version}'; reconnect_backoff_ms (int) – The amount of time in milliseconds to wait before attempting to reconnect to a Delete topics from the cluster. You can set different values for different topics. Delete the whole topic. It's a kind of tricky and I don't recommend this way. Create a new topic for every day. Something like my-topic-2015-09-21. But I don't think you need to delete the messages in the topic at all. Because your Kafka consumer keeps track of messages that has been already


Topic Deletion · The Internals of Apache Kafka, Use kafka-topics.sh --delete to delete remove-me topic. +. $ ./bin/kafka-topics.sh --zookeeper localhost:2181 --delete --topic remove  You can programmatically create topics either using kafka-python or confluent_kafka client which is a lightweight wrapper around librdkafka.


How can I delete all the messages from a Kafka topic?, I created a Kafka topic and pushed large messages to that kafka topic Jun 4, 2019 in Apache Kafka by sarvesh To verify this, try running the Confluent develops and maintains confluent-kafka-python, a Python Client client for Apache Kafka® that provides a high-level Producer, Consumer and AdminClient compatible with all Kafka brokers >= v0.8, Confluent Cloud and Confluent Platform. Python Client installation. The client is available on PyPI and can be installed using pip: