push multiple Line Text as one message in a kafka topic

kafkacat
kafka-console producer
kafka tool send message
kafka tail topic
kafka console-producer from file
kafka consumer from beginning
kafka check if topic exists
kafka-console-producer send json

I want to push a text consisting of multiple lines as one message into a kafka topic.

After I enter:

kafka-console-producer --broker-list localhost:9092 --topic myTopic

and copy my text:

My Text consists of:
two lines instead of one

I get two messages in the kafka topic, but I want to have just one. Any ideas how to achieve that? Thanks

You can use kafkacat for this, with its -D operator to specify a custom message delimiter (in this example /):

kafkacat -b kafka:29092 \
        -t test_topic_01 \
        -D/ \
        -P <<EOF
this is a string message 
with a line break/this is 
another message with two 
line breaks!
EOF

Note that the delimiter must be a single byte - multi-byte chars will end up getting included in the resulting message See issue #140

Resulting messages, inspected also using kafkacat:

$ kafkacat -b kafka:29092 -C \
         -f '\nKey (%K bytes): %k\t\nValue (%S bytes): %s\n\Partition: %p\tOffset: %o\n--\n' \
         -t test_topic_01

Key (-1 bytes):
Value (43 bytes): this is a string message
with a line break
Partition: 0    Offset: 0
--

Key (-1 bytes):
Value (48 bytes): this is
another message with two
line breaks!

Partition: 0    Offset: 1
--
% Reached end of topic test_topic_01 [0] at offset 2

Inspecting using kafka-console-consumer:

$ kafka-console-consumer \
    --bootstrap-server kafka:29092 \
    --topic test_topic_01 \
    --from-beginning

this is a string message
with a line break
this is
another message with two
line breaks!

(thus illustrating why kafkacat is nicer to work with than kafka-console-consumer because of its optional verbosity :) )

Sending multiline messages to Kafka, You can use kafkacat to send messages to Kafka that include line breaks. Partition: 0 Offset: 1 -- % Reached end of topic test_topic_01 [0] at  The KafkaConsumer node then receives messages that are published on the Kafka topic, as input to the message flow. You can use a KafkaProducer node to publish messages from your message flow to a topic that is hosted on a Kafka server. For more information about using a KafkaProducer node, see Producing messages on Kafka topics.

It's not possible with kafka-console-producer as it uses a Java Scanner object that's newline delimited.

You would need to do it via your own producer code

Sending some messages from the console, Kafka installation has a command-line utility that enables you to produce data. next command followed by some text that will be sent to the server as messages: The earlier inputs will push two messages to the kafkatest topic present in the --sync : This specifies that the messages should be sent synchronously—one at  kafkacat -b kafka:29092 \ -t test_topic_01 \ -D/ \ -P <<EOF this is a string message with a line break/this is another message with two line breaks! EOF Note that the delimiter must be a single byte - multi-byte chars will end up getting included in the resulting message See issue #140

With Console-consumer you are obviously running tests for your expected data coming from client. If it is a single message, better keep it as a single string by adding a unique delimiter as identifier. e.g.

{this is line one ^^ this is line two}

Then handle the message accordingly in your consumer job. Even if client is planning to send multiple sentences in message, better make it in a single string, it will improve serialization of your message and will be more efficient after serialization.

Documentation - Apache Kafka, With MirrorMaker, messages are replicated across multiple datacenters or cloud Since we have only one partition for this topic there is only one line. source data system into Kafka or push from Kafka into some sink data system. Also currently, master key/secret is stored as plain text in server.properties config file. With Kafka Connect, writing a topic’s content to a local text file requires only a few simple steps. Starting Kafka and Zookeeper. The first step is to start the Kafka and Zookeeper servers. Check out our Kafka Quickstart Tutorial to get up and running quickly. Creating a Topic to Write to. Creating a topic from the command line is very easy to do.

kafkacat Utility, A traditional queue retains records in-order on the server, and if multiple Since we have only one partition for this topic there is only one line. input topic ""​streams-file-input", where message values // represent lines of text (for the system or application into Kafka or push from Kafka into some sink system or application. ~/kafka-training/lab1 $ ./start-consumer-console.sh Message 4 This is message 2 This is message 1 This is message 3 Message 5 Message 6 Message 7 Notice that the messages are not coming in order. This is because we only have one consumer so it is reading the messages from all 13 partitions.

Kafka Tutorial: Using Kafka from the command line, Each partition has one server which acts as the "leader" and zero or more Kafka also has a command line consumer that will dump out messages to standard out. send data to multiple topics * @param messages list of producer data objects shared by most messaging systems, where data is pushed to the broker from  Yes you can control compatibility as @markdav suggests.. Re: subjects, the schema registry is not tied only to use with Kafka, so we use the name "subject" to express a series of related schemas, where "related" generally means you'll find messages with different versions possibly mixed together in the same dataset.

How to Implement a Kafka Producer, kafkacat is a command line utility that you can use to test and debug Apache Kafka® deployments. In consumer mode, kafkacat reads messages from a topic and partition and prints For example, to consume a single message: Run it with the -P command and enter the data you want, and then press Ctrl-D to finish:. In this respect Kafka follows a more traditional design, shared by most messaging systems, where data is pushed to the broker from the producer and pulled from the broker by the consumer. Some logging-centric systems, such as scribe and flume follow a very different push based path where data is pushed downstream.

Comments
  • It's better not to rely on random delimiters within messages at all, though
  • Agree, in fact if test case contains having multiple lines, use "\n" delimiter to simulate this scenario. e.g. {this is line one \n this is line two}