How does Azure Service Bus identify a duplicate message?

azure service bus duplicate detection messageid
azure service bus acknowledge
azure service bus queue
service bus vs message queue
service bus message label
azure service bus patterns
azure service bus push or pull
azure queue vs service bus

I understand that Azure Service Bus has a duplicate message detection feature which will remove messages it believes are duplicates of other messages. I'd like to use this feature to help protect against some duplicate delivery.

What I'm curious about is how the service determines two messages are actually duplicates:

  • What properties of the message are considered?
  • Is the content of the message considered?
  • If I send two messages with the same content, but different message properties, are they considered duplicates?

The duplicate detection is looking at the MessageId property of the brokered message. So, if you set the message Id to something that should be unique per message coming in the duplicate detection can catch it. As far as I know only the message Id is used for detection. The contents of the message are NOT looked at, so if you have two messages sent that have the same actual content, but have different message IDs they will not be detected as duplicate.


MSDN Documentation:

If the scenario cannot tolerate duplicate processing, then additional logic is required in the application to detect duplicates which can be achieved based upon the MessageId property of the message which will remain constant across delivery attempts. This is known as Exactly Once processing.

There is also a Brokered Message Duplication Detection code sample on that should be exactly what you are looking for as far as proving it out.

I also quickly tested this out and sent in 5 messages to a queue with RequiresDuplicateDetection set to true, all with the exact same content but different MessageIds. I then retrieved all five messages. I then did the reverse where I had matching MessageIds but different payloads, and only one message was retrieved.

Azure Service Bus duplicate message detection, Duplicate message detection on a queue or topic is configured when the messaging entity is created by specifying properties in the appropriate description class  I understand that Azure Service Bus has a duplicate message detection feature which will remove messages it believes are duplicates of other messages. I'd like to use this feature to help protect against some duplicate delivery.

In my case I have to apply ScheduledEnqueueTimeUtc on top of MessageId. Because most of the time the first message already got pickup by worker, before the sub-sequence duplicate message were arrive in the Queue. By adding ScheduledEnqueueTimeUtc. We tell the Service bus to hold on the the message for some time before letting worker them up.

            var message = new BrokeredMessage(json)
                MessageId = GetMessageId(input, extra)

            // Delay 30 seconds for Message to process
            // So that Duplication Detection Engine has enought time to reject duplicated message
            message.ScheduledEnqueueTimeUtc = DateTime.UtcNow.AddSeconds(30);

Configuring Duplicate Message Detection, In Service Bus brokered messaging we are able to create idempotent If the queue or topic that the message is being sent to is configured to detect duplicate​  How to check message is duplicate using Text in Azure Service Bus Queue instead if MessageId. I tried MSDN CODE and properties like . namespaceManager.CreateQueue(new

Another important property to be considered while dealing with 'RequiresDuplicateDetection' property of a Azure Service Bus entity is 'DuplicateDetectionHistoryTimeWindow', the time frame within which message with duplicate message id will be rejected.

Default value of duplicate detection time history now is 30 seconds, the value can range between 20 seconds and 7 days.

Enabling duplicate detection helps keep track of the application-controlled MessageId of all messages sent into a queue or topic during a specified time window. If any new message is sent carrying a MessageId that has already been logged during the time window, the message is reported as accepted (the send operation succeeds), but the newly sent message is instantly ignored and dropped. No other parts of the message other than the MessageId are considered.

Handling Duplicate Messages, When duplicate detection is enabled, Azure Service Bus will start identifying the duplicate messages. This period can be configured to range  You can use the following protocols with Azure Service Bus to send and receive messages: Advanced Message Queuing Protocol (AMQP) Service Bus Messaging Protocol (SBMP) HTTP; See the following table for the outbound ports you need to open to use these protocols to communicate with Azure Event Hubs.

Avoid Message Loss While Resubmitting to Azure Service Bus, involving message resubmission to an Azure Service Bus entity with 'Requires Duplicate Detection' property enabled, and the requirement is  Microsoft Azure Service Bus is a fully managed enterprise integration message broker. Service Bus can decouple applications and services. Service Bus offers a reliable and secure platform for asynchronous transfer of data and state. Data is transferred between different applications and services using messages. A message is in binary format and can contain JSON, XML, or just text.

Message Resubmitting challenge with duplicate detection enabled , It would be great if we could have duplicate detection based on the current content of the Queue. E.g.. If there is a message with a message-id '  Azure Service Bus employs multiple message brokers to process messages and multiple messaging stores to store messages. A conventional queue or topic is handled by a single message broker and stored in one messaging store. Service Bus partitions enable queues and topics, or messaging entities,

Detect Duplicate messages in the service bus , The sample is specifically crafted to demonstrate the effect of duplicate detection when enabled on a queue or topic. The default setting is for duplicate detection to​  In essence you'll be broadcasting the message to all of your services. Thanks to competing consumer, only one instance of each service will get that message. To target a specific service, you'd create an additional rule that would filter out message on a property (a standard property or a custom one). For example, it could be a service name.

  • Do you have any references that would support your answer? I struggled to find anything concrete - a code example would suffice.
  • Sure: I'll add detail to the answer.
  • When detecting duplicates, does the queue keep the oldest message or the newest message?
  • @Codegrue It simply ignores the second, or subsequent messages. Only the first is kept. Depending on the length of the window of the duplicate message history and the length of the queue, indeed the first message may have already been processed.
  • Make a hash of the message text and assign it as message id.