In Disassembler pipeline component - Send only last message out from GetNext() method

I have a requirement where I will be receiving a batch of records. I have to disassemble and insert the data into DB which I have completed. But I don't want any message to come out of the pipeline except the last custom made message.

I have extended FFDasm and called Disassembler(), then we have GetNext() which is returning every debatched message out and they are failing as there is subscribers. I want to send nothing out from GetNext() until Last message.

Please help if anyone have already implemented this requirement. Thanks!

If you want to send only one message on the GetNext, you have to call on Disassemble method to the base Disassemble and get all the messages (you can enqueue this messages to manage them on GetNext) as:

 public new void Disassemble(IPipelineContext pContext, IBaseMessage pInMsg)
 {
    try
    {    
       base.Disassemble(pContext, pInMsg);

       IBaseMessage message = base.GetNext(pContext);
       while (message != null)
       {    
           // Only store one message                   
           if (this.messagesCount == 0)
           {
              // _message is a Queue<IBaseMessage>
              this._messages.Enqueue(message);

              this.messagesCount++;
           }

           message = base.GetNext(pContext);
       }              
    }
    catch (Exception ex)
    {
       // Manage errors 
    }

Then on GetNext method, you have the queue and you can return whatever you want:

public new IBaseMessage GetNext(IPipelineContext pContext)
{
   return _messages.Dequeue();
}

Microsoft BizTalk Server 2010 Unleashed, Even then for any reason if you have to insert from pipeline component then do the following: Please note, GetNext() method of IDisassembler interface is not  It differs from the standard Flat File Disassembler in that it does not produce any disassembled documents until the entire input interchange is completely processed. This component implementation inherits from the FFDasmComp class and overrides the GetNext method.

The recommended approach is to publish messages after disassemble stage to BizTalk message box db and use a db adapter to insert into database. Publishing messages to message box and using adapter will provide you more options on design/performance and will decouple your DB insert from receive logic. Also in future if you want to reuse the same message for something else, you would be able to do so.

Even then for any reason if you have to insert from pipeline component then do the following:

Please note, GetNext() method of IDisassembler interface is not invoked until Disassemble() method is complete. Based on this, you can use following approach assuming you have encapsulated FFDASM within your own custom component:

Insert all disassembled messages in disassemble method itself and enqueue only the last message to a Queue class variable. In GetNext() message then return the Dequeued message, when Queue is empty return null. You can optimize the DB insert by inserting multiple rows at a time and saving them in batches depending on volume. Please note this approach may encounter performance issues depending on the size of file and number of rows being inserted into db.

Extending the Flat File Disassembler Pipeline Component, using a send pipeline with your custom assembler component inside an orchestration The first method is called Disassemble. It is called with the input message to the pipeline component, and it is only The second method is called GetNext. This allows the disassembler to debatch the incoming message and return  GetNext. Returns a single message from the Message set, that was stored in Disassmble() method call. This method will be called until it returns Null. Accepts IPipelineContext as input parameter, whereas IPipelineContext is the context of executing pipeline. Returns IBaseMessage from the message set.

I am calling DBInsert SP from GetNext()

Oh...so...sorry to say, but you're doing it wrong and actually creating a bunch of problems doing this. :(

This is a very basic scenario to cover with BizTalk Server. All you need is:

  1. A Pipeline Component to Promote BTS.InterchageID
  2. A Sequential Convoy Orchestration Correlating on BTS.InterchangeID and using Ordered Delivery.
  3. In the Orchestration, call the SP, transform to SOAP, call the SOAP endpoint, whatever you need.
  4. As you process the Messages, check for BTS.LastInterchagneMessage, then perform your close out logic.

To be 100% clear, there are no practical 'performance' issues here. By guessing about 'performance' you've actually created the problem you were thinking to solve, and created a bunch of support issues for later on, sorry again. :( There is no reason to not use an Orchestration.

As noted, 25K records isn't a lot. Be sure to have the Receive Location and Orchestration in different Hosts.

Custom BizTalk Pipeline Disassembler Component, Sign out. BizTalk · Why BizTalk; Docs. BizTalk · HIS · Logic Apps Extending the Flat File Disassembler Pipeline Component On the first call to the GetNext method, it processes all messages in the interchange, stores The implementation of the GetNext() method in the code sample below would not be  A disassembling pipeline component receives one message on input and produces zero or more messages on output. Disassembling components are used to split interchanges of messages into individual documents. Disassembler components must implement the following interfaces: IBaseComponent. IDisassemblerComponent. IComponentUI. IPersistPropertyBag .

UnzipDisassembler, It can also be encrypted and digitally signed before sending it out on the wire. The send pipeline consists of four stages: Preassemble: As the  The first one tells that this is a pipeline component and the second one restricts the component to be used only in the disassemble stage in a pipeline. The specific interfaces a pipeline component implements are what differentiate that pipeline component from another.

BizTalk Pipeline Custom Component Message Unzippper – Datavoid, This article shows how to create a custom pipeline component in Our component will receive a ZIP file, uncompress it, and send its public UnzipDisassemblerComponent() { // // TODO: Add constructor This method is responsible for returning the messages to BizTalk, Question about getNext() Pin. 1 Answer 1. Splitting messages can only be done with a disassembler component. You can create a class that inherits from an existing disassembler (like what Selvin did) or you can specify that you want to create a "DisassemblingParser" component type for receive pipeline type in the Pipeline Component Wizard.

Pipeline Component Best Practices and Examples, This Custom Pipeline Component for BizTalk Server 2010, receives a easily send one compressed file (with all the ten files in it), taking as only Now drag this component to the Dissamble fase andadd a XML Disassembler right Have the GetNext() method to enforce the pipeline to send each one of  The envelope schemas as well as the individual document schemas are associated with the disassembler component in Pipeline Designer. The XML Disassembler only processes data in the body part of the message. Thus, only properties from body part can be promoted.

Comments
  • Sorry, if you're inserting into a database, how are you not processing the batch messages themselves?
  • Showing the relevant code in the question would help people answer it.
  • This sounds to be the solution. Let me try this, I will get back to you. Thanks!
  • MR. felixmondelo, My sincere thanks to you. This worked amazingly and that too we scored record performance by inserting 25K data to Table in less than a minute. Previously it took around 7 mins. That is simply superb. Your solution made it possible. A BIGGGG CHEERS to you!!!
  • For the benefit of future readers, DO NOT DO THIS. It's a very risky and fragile pattern that eliminates much of the benefits of using BizTalk. There are several better ways to accomplish this, even the <1 min performance.
  • Hi Vikas, as of now, I am calling DBInsert SP from GetNext() as I get each individual record in GetNext(). My volume is huge about 25K flat file records which needs to be stored in DB first. With last message I trigger orch to poll each record and convert to SOAP request.
  • Hi Vikas, as of now, I am calling DBInsert SP from GetNext() as I get each individual record in GetNext(). My volume is huge about 25K flat file records which needs to be stored in DB first. With last message I trigger orch to poll each record and convert to SOAP request. By calling SP in Disassembler() it will hit performance badly. That's why I am doing in GetNext() but for each GetNext() there is a dummy message sent out which I want to avoid. Is there any way to do this?
  • You should rather use a different approach, 25k is not a huge volume. You can let these messages publish to message box and from there you can send it to MSMQ. And then you can do whatever u want with them. MSMQ can be used on any BizTalk server you have and is part of Windows. You can use MSMQ adapter to send or receive messages from a queue and process them to go to both DB as well as create SOAP request.
  • No Vikas, that wont be accepted. This is the Architecture provided and I have to follow that. So there is no way to stop GetNext() to publish empty message to MsgBox?
  • Hi Johns, My requirement is such that I have to call SP from pipeline to avoid any loss of data. 1. Load data to DB 2. Create soap request for each record. 3. Get the SOAP response. 4. Batch all the responses and send to external system as FF.
  • Ok...even worse. Sorry, but you're creating potentially catastrophic gaps by doing this. Meaning, you're creating a much greater probability of data loss by calling the SP in the Pipeline vs using all the reliability features of BizTalk. Using BizTalk as I describe is completely reliable and, to be blunt, much more reliable than the pattern you have implemented already.
  • Everything you describe should be done with an Orchestration and DB Adapters where you have built in retry, error handing etc. Otherwise, you will have to basically re-write all this yourself.
  • The orch approach has a problem. 1. You need one orch just to call SP with retry and error handling. 2. you need one more orch to poll the DB, create request response. I want to eliminate the 1st orch by calling SP in DB and dont want that debatched message to publish to msgBox.
  • But those aren't "problems". They're just not. You can do all of that in one Orchestration if you want. Though I would probably use two, maybe three called in succession. There is nothing wrong with this, nothing. In fact, that is the correct way.