Featured

Multi root node – handling credit notes and invoices as one message

Recently i came across a requirement where invoices must be split in either credit note or invoice. This is in it self not anything special except that the different message types did not share a common root node.

So instead of having a common root node like

You have two files like

Biztalk can only handle one root node in a map either on source or destination (let’s disregard orchestrations for the moment). We also assume here that our application sends out invoices/credit notes in the same message structure and the only way to see if it’s an invoice or credit note is to inspect the value of payable amount (Invoice >= 0 and credit note < 0).


There is several solutions to this

  • Create an orchestration that splits the incomming message depending on the value of payable amount.
    My thought was that using an orchestration for this was unnecessary and you will need two map’s plus you need to promote payable amount.
  • Create two adapter’s that has a filter on payable amount.
    You still need two map’s plus you need to promote payable amount.
  • Use an external xslt file. This is of course a valid solution, and would solve all my problems, but i like to use the mapper as it’s easier to maintain and strangely enough there are a lot of Biztalk developers that don’t know XSLT which is really crippling.
  • My solution.

    1) Create a dummy schema that contains a dummy root node plus the two schemas underneath.


    2) Create a send pipelinecomponent that uses a stream class to ignore the top root node and copy all the dummy root node’s attributes,that does not have a namespace prefix that starts with ns[\d], to the new root node.
    At the same time i made it possible to add an schemalocation and remove all namespace prefixe’s that start with ns[\d].

    You must of course make sure that you only create one of the invoice types.

The source code is located here

ScheduledTask Adapter – Monthly

When trying to set up a schedule for each month for a year we ran into a known bugg in the ScheduledTask Adapter.

The Messaging Engine failed to add a receive location “xxx” with URL “schedule://Monthly/xxx” to the adapter “ScheduleAdapter”. Reason: “Time-out interval must be less than 2^32-2.
Parameter name: dueTime”

Apparently, Microsoft.BizTalk.Scheduler.Runtime.dll (BizTalk DLL) doesn’t accept schedule a task with more than 50 days.

This post shows a work arround for the one fortunate to have the latest code for the ScheduleTask adapter.

Well you cannot get arround the fact that Microsoft.BizTalk.Scheduler.Runtime.dll does not allow more then 50 days and i did not want to change to much in the code.

My solution was to make make the adapter trigger each weekday selected or calculated by user selection.
I then check if the weekday exists in any of the months selected.

Update the MonthSchedule class

First you need to update the method GetNextActivationTime in the MonthSchedule class.

Instead of calculating and returning exact month and day, one should return nearest month with the specified weekday.

ScheduledEndpoint class calls the GetNextActivationTime  method in the MonthSchedule class.

Either GetNextActivationTime  returns nearest month with selected weekday

Monthly_day

or GetNextActivationTime  returns nearest month with calculated ordinal weekday

Monthly_ordinal

Update the EndpointTask class

EndpointTask method in the ScheduledEndpoint class is now executed every month and checks if the current month exists in the months selected.

EndpointTask

Happy coding

Refresh dependent CSS elements

I’m been doing some WEB programming and i wanted sibling elements to show or hide depending on if a specific attribute of a parent had a specific value or not.

Using this CSS

*[data-json-dirty=”yes”] .json-save {
position:absolute;
display: inline-block;
}

*[data-json-dirty=”no”] .json-save {
display: none;
}

.json-save:before {
content:”\2713″;
}

and this markup

<table style=”width:100%”>
<tbody data-json-dirty=”no” class=”none” onclick=”tooglesaveState(this);”>
<tr><td>

<input type=”text” style=”width:100%” value=”mamma” >Test</td></tr>
</tbody>
</table>

and this javascript

function tooglesaveState(obj)function tooglesaveState(obj){ if(obj.dataset.jsonDirty==’no’)
{
obj.dataset.jsonDirty=’yes’;
}
else{
obj.dataset.jsonDirty=’no’;
}

obj.className = obj.className;
}

I hide the DIV element with the style class json-save.

The magic lies in the last statement in the javascript

obj.className = obj.className

The above statement will refresh the CSS specified for the element and all dependent elements.

BizTalk ManagementService – problems with dot (.) in REST requests

I’ve been working some with the BizTalk ManagementService and one thing i run into is that searching for a specific item that has a dot (.) in it does not work.

So if you use the standard Swagger UI to do the request bellow

pipe1

You will get a , 404 – Not Found , HTTP response.

The solution is simple, add a slash in the request like bellow

pipe2

The GUI adds %2F instead but eiher way works

 

Customizable, custom BizTalk send port macros

Why custom macros

Macros on a send port is a neat feature, but it is too little, it cannot handle needs that are more specific.

Today we use the “standard” way of changing the system context property ReceivedFileName in a pipeline component and then using this via the %SourceFileName% macro.

This is of course a solution but this makes it hard for none initiated persons to really know what value to expect.

Description

The component, which must be used on a send pipeline, uses RegEx to extract all custom macros from the filename specified on the adapter configuration. It also uses RegEx to extract custom parameters.

Most of the custom macros, allows you to add custom input.

Available macros

%DateTime(?)%
%ReceivedDateTime(?)%
%FileDateTime(?)%
%FilePattern(?)%
%FileNameOnly%
%Context(?)%
%Folder%

DateTime

The %DateTime(?)% macro allows you to use use any part of the current DateTime. In short the component uses DateTime.Now.ToString(?), and the input to this macro is used as input to the ToString function.

Examples

%DateTime(%d)%, current day number

%DateTime(yyyy)%, current year

%DateTime(MM)%, current month

The component does not check to see if it is a valid format.

ReceivedDateTime

%ReceivedDateTime(?)% macro works exactly as DateTime macro, with the difference that the date is picked from system context property AdapterReceiveCompleteTime instead.

AdapterReceiveCompleteTime is the date and time when BizTalk picked up a specific message.

Remember that tracking must be enabled for this property to be present.

FileDateTime

%FileDateTime(?)% macro works exactly as DateTime macro, with the difference that the date is picked from system context property FileCreationTime

This is the date when the incomming file was created on disk. Should also work with FTP but I have not tested it.

FilePattern

%FilePattern(?)% macro allows you to use a RegEx expression on the Received filename. What the component does is to return the group items found on an expression match (please read up on RegEx if this is confusing)

Example

If we take the macro expression %FilePattern(^.{3}(.{3})([0-9]+))% and apply it to the file name NOTYES12345Exclude.xml, we would get the returned result YES12345.xml

That is because the expression have 2 groups specified (.{3}) and ([0-9]+). The first group extracts the 3 first chars and the second expression extracts all numbers after these 3 initial chars. Everything else is excluded.

TIP. If you wanted to switch the location, you could use different macros with different expressions.

Context

The %Context(?)% macro allows you to use a any context property value. To make the filename as small as possible only prefixes are used/allowed. So instead of using something like %Context(http://schemas.microsoft.com/BizTalk/2003/system-properties#Property)% we use the abbreviation %Context(BTS.Property%)

The following standard context abbreviations are possible

SBMessaging http://schemas.microsoft.com/BizTalk/2012/Adapter/BrokeredMessage-properties
POP3 http://schemas.microsoft.com/BizTalk/2003/pop3-properties
MSMQT http://schemas.microsoft.com/BizTalk/2003/msmqt-properties
ErrorReport http://schemas.microsoft.com/BizTalk/2005/error-report
EdiOverride http://schemas.microsoft.com/BizTalk/2006/edi-properties
EDI http://schemas.microsoft.com/Edi/PropertySchema
EdiIntAS http://schemas.microsoft.com/BizTalk/2006/as2-properties
BTF2 http://schemas.microsoft.com/BizTalk/2003/btf2-properties
BTS http://schemas.microsoft.com/BizTalk/2003/system-properties
FILE http://schemas.microsoft.com/BizTalk/2003/file-properties
MessageTracking http://schemas.microsoft.com/BizTalk/2003/file-properties

There is also a possibility to use one (1) custom prefix. I have used the prefix CST for all none Microsoft properties.

So if you have a custom context property you want to use, use the following syntax.

%Context(CST.Property%)

What the component will do when it finds a custom context macro is to go through all context properties that are not MS specific and return the value from the first property found with the name specified.

This means of course that this is a little bit riskier, you could possible pick up the wrong value. So only use this if you know that your context name is unique amongst the other none MS context properties.

Folder

Folder is a simple but helpful macro as a common requirement is to put a message in a specific folder depending on a date. Let’s say a file was created 2017-01-15. Then the requirement would be that the file should be saved in a structure like bellow.

Year
Month
∟∟ Day

The folder macro simply inserts a back slash in its place. The component makes sure that the folder structure exists or creates it if it does not.

Example

If the URI is c:\integrations\int0010\ and the current date is 2017-05-04 and the macro expression %DateTime(yyyy)%%Folder%%DateTime(MM)%%Folder%%DateTime(dd)%%Folder%message.xml is used, then the final folder location would be c:\integrations\int0010\2017\05\04.

This of course does only work for the file adapter.

The code for the pipeline and pipeline component can be downloaded from here

Should i place the map on the receive or send port?

For you BizTalk newbies

I have the fortunate pleasure of having two complete fresh junior developers under my wings.

Today i got a question if a map should be placed in the receive or send port, in this case these were the two only logic choices.

For me there exists an obvious logic way to make the right decision and with this post i want to share my thoughts in the matter, maybe helping somebody thinking of the same thing and as always hoping for some creative feedback.

My logic comes from my golden rule “Use as little resources as possible”, in this case disk space used when saving the message in the message box.

If the message is smaller after mapping the map should be placed in the receive port

Better to save the smaller message in the messagebox which means less space in the SQL server. What many people forget or don’t know is that all messages is saved twice, first in the SQL server database and second in the SQL log file, then we are not even talking about possible tracking.

If the message is bigger after mapping put it in the send port

This means that the bigger message will never be saved in the messagebox as long as tracking is not enabled.

Promoted properties

There does of course exist exceptions to this rule one being if the mapping promotes a property used for routing.

You should always think about which is the best place to create your promoted properties , in the source or the target schema, and try to place them according to the rule i have already stated.

Happy coding

Adding keep file(s) in blogical sftp adapter

Improving blogical sftp adapter

One of my customers wanted to pick up a file from a partner sftp site, the problem being, that the partner did not allow you to delete the file.

The reason being that the partner had multiple consumers of the file and they did not want to recreate it for every partner.

With BizTalk 2013 R2 you now have the possibility to use “Delete After Download” and “Timestamp comparison” with the FTP adapter.

Strangely enough this does not exist in the sftp adapter. Sadly enough it did not exist in the blogical sftp adapter either.

At least with the  blogical sftp adapter you have the source code and the signing key.

With this and with some help from dotPeek (JetBrains) i was able to add the functionality  to existing blogical sftp.

I did not want to add new properties so i gave a new meaning to the get action enum DoNothing. Which i find totally dangerous, well that means if your intent is to create an endless loop or not.

In my new implementation “after get action” DoNothing or “blank” in the admin window now means keep file and keep a reference to the filename.

Instead of creating something totally new i used what was already in place for the standard FTP adapter, i used existing tables and procedures and stole much of the plumbing from the FTP adapter.

The adapter uses the table [BizTalkMsgBoxDb].[dbo].[adap_DownloadedFiles] to keep a record of downloaded files.

I also removed the batch functionality when using the DoNothing action, as that could or would again create an endless loop.

I did not change all the code as i only wanted to create a quick fix.

But at least i want to share what i have done as i think other people may have the same problem.

You can get the updated dll file Blogical.Shared.Adapters.Sftp.dll from this address https://1drv.ms/u/s!ALZwtKGCKVWcmyg.

All you have to do is replace it with the existing file in the folder %program files%\Blogical\Blogical.Shared.Adapters.Sftp.

Also you will have to restart all hosts connected to this adapter.

 

 

 

 

BizTalk property demotion

Almost everybody knows what property promotion is, property demotion however is not that well known.

Property demotion is the opposite of promoting something from the message to context, demotion means copying a value from context to an outgoing message.

Demotion is usually a way to add data to the envelope on outgoing messages as there is no other standard way to add dynamic values to an envelope. It is however also possible to use demotion also on the outgoing document.

To make the magic work you have to do the following

  • First of all the property must exist in context. So you must either promote or write a value to context or make sure the property you want to use is promoted in the schemas used in the incoming messages.
    The context property does not need to be promoted but it must exist in context. The sample uses a custom pipeline to promote the date and time when the message entered the pipeline. In a real life scenario you would make sure that the date time is fetched before the assembler component, in the decode stage, and not after as I’m doing in the validate stage as assembler processing can take some time.
  • The element you want to populate in the message must be promoted in the schema with the exact property name and property namespace. The context property type must also be of the same type as the element you want to populate. You can use properties both from standard schemas as well as custom schemas. The sample uses both.
  • The element to be populated must exist in the message that is processed in the send pipeline and the element must be empty. This is very easy to do in a map, just set the value property to <empty> in the map.
  • Use XmlTransmit pipeline in the send port as there is where the demotion happens.

You can download the BizTalk 2010 sample from my one drive

That’s it, Happy coding.

Recursive loop of self referencing record

Recursive loop of self referencing record

In this post i will show you how to map a self referencing record. A self referencing record is a schema type that references it self and in this way can have never ending structure like in the picture bellow.

selfref

I solved this by using a xslt template and apply-templates

 RecTemplate

I use a dummy param to get the correct context, see bellow, the active node in this case becomes the second Routing record that the link goes from.

RefContext

Remember the template must have a match attribute targeting the Routing element. When you use the apply-templates inside another template it will process all Routing element’s inside the active node.

This is like a for-each on steroids 🙂

Pattern count stream

I resently got a question on linkedin about using streaming approach in BizTalk pipelines. More precisely how to read a part of the message without reading the complete message in memory.

The underlying goal was to get the message count in a batch message, with envelope . Strangely enough xmlreceivepipeline does not return the message count. The component only returns the index of the specific message plus that the last message is marked as the last message in the interchange.

What i created is a none specific message stream as i wanted it to handle both text and xml messages. For this to be possible a pattern search is made in a streaming fashion of course.
Even though the example uses a custom xml disassembler component the custom stream in it self can be used by any disassembler component.
ext_prop
To count the messages in the batch in a streaming fashion you must replace the original message with the new custom message using the original message as input.
ext_diss
What happens then is that when the disassembler reades the message it realy reades the new custom stream. The stream returns the message as is without modifiyng it in any way but it keeps a count of how many times a specific pattern occurs in the original stream.
ext_code1
As the message is read in chunks by the disassembler the search is done byte by byte. The xml disassembler reads the message two times. Once when searching from promoted properties and the second time when the messages are returned by the GetNext method.
I make sure that the stream only does a pattern search the first time the pipeline component traverses the message.
ext_code2
The pattern match count is save in a private variable called recordCount that is later promoted on each message that is returnedby the GetNext method.
ext_code3
The sample includes a property schema with the specified namespace and property name.

So in a nutschell the biztalk streaming pattern is as follows

  • Replace the original biztalk message stream with a custom stream.
  • Replace the BodyPart.Data with your custom stream.
  • When biztalk now reads the stream as it possible does several times before it reaches the messagebox it actually reads your custom stream which in turn reads the original biztalkmessage.

The custom stream can then do anything with the original stream data even changing the content before it returns it to biztalk.
What you gain by doing this is

  • The message dones not need to be traversed more times then neccesarry.
  • Low memory consumtion.

The code for the solution can be found in my one-drive in a zip file called ExtendeDisassemblers.zip

BizTalk enrichment pattern without Orchestration

One common pattern in BizTalk is to enrich a message with information that it does not have. For example if you receive a customer update from your web site and you need to update the ERP system.

Problem is the web site does not know what the customer number is in the ERP system, it only knows the web customer number.

In this scenario one usually has a side system in sql server containing a mapping between the two numbers. The common way to solve this is to create an orchestration that calls the sql server to retrieve the ERP customer number, the last step in the orchestration is to map the sql response with the original web message.

As the title shows my solution does not use an orchestration, instead it uses a WCF behavior.

The WCF behavior has two properties

  • MapName. Here you must add the full name of the map. That means full name of the map + the name of the assembly.
  •  Namespace. Add the namespace that the new .merged, message should have. The root node is always RequestResponse in this implementation.

Enrich_BehaviorProperties

What we do is to intercept the request message before it is sent to sql server in the BeforeSendRequest method.

Enrich_BeforeSendRequest

The method TransformMessage executes the BizTalk map which you would normally do in the send port, but as we want to save the original message we execute it in the BeforeSendRequest methodinstead.

I have also created a custom stream that allows you to stream an XmlDictionaryReader as the class BTSXslTransform , Transform method, only takes stream as input.

I have written an earlier post on how to execute a BizTalk map in BizTalk pipeline component, you can read it at   https://fernandodosanjos.wordpress.com/2014/07/16/executing-biztalk-map-in-pipeline-with-arguments/.

The original message, not the sql request, is returned in the BeforeSendRequest method which is later on available in the AfterReceiveReply method.

Enrich_AfterReceiveReply

The AfterReceiveReply is executed when the response, reply, message is returned, the parameter correlationState contains, in this case, the original message.

Here we merge the original message with the response message that could either be a valid sql response message or a fault message.

I have written an earlier post on how to merge request and response messages similar to this post, you can read it at http://code.msdn.microsoft.com/Wcf-merge-request-response-37e7801b. The post explains how you create a schema for the merged messages and how to handle fault messages.

Download code

As always i hope for some feedbackon how to improve the solution.

Happy coding