ALIGNING MICROSOFT AZURE BIZTALK SERVICES DEVELOPMENT WITH ENTERPRISE INTEGRATION PATTERNS

We have just finished a fairly large effort in moving the QuickLearn Team blog over to http://www.quicklearn.com/blog, as a result we had a mix up with the link for our last post.

This post has moved here: Aligning Microsoft Azure BizTalk Services Development with Enterprise Integration Patterns

What’s New In BizTalk Server 2013 R2

This is the first in a series of posts exploring What’s New in BizTalk Server 2013 R2. It will also serve as the index of the series, and contain links to all of the posts to come.


This is a listing of all of the posts within the series:

  1. What’s New In BizTalk Server 2013 R2
    Shows Shared Access Signature (SAS) Authentication for Service Bus
  2. Getting Started with BizTalk Server 2013 R2’s Built-in Health Monitoring
    Demonstrates the installation and use of the BizTalk Health Monitor
  3. JSON Schemas in BizTalk Server 2013 R2 [Code Sample]
    Shows how to generate a JSON schema and write unit tests to validate instances
  4. Decoding JSON Data Using the BizTalk Server 2013 R2 JsonDecode Pipeline Component [Code Sample]
    Shows how to receive JSON messages and write integration tests to validate a configurable pipeline

We’ve been pretty busy over here at QuickLearn over the past few months, as many of you may have noticed. We’ve released our BizTalk Server 2013 Administrator Deep Dive class, and have been hard at work on our Azure BizTalk Services Fundamentals class (coming as soon as September 2014). Meanwhile, Microsoft has released BizTalk Server 2013 R2.

As a result, I am starting a series in a similar vein as my What’s New in BizTalk Server 2013 series, to uncover those new features in 2013 R2 that will make your life just a little bit easier. However, this time around it will be a weekly series that will occasionally take breaks to share time with posts about Azure BizTalk Services.

All of that having been said, I’m going to get upgraded, and then jump right in to check out one of the things I’m most excited about.

image

I Love Microsoft Azure Service Bus

I’ve got to admit that I’m a huge fan of Microsoft Azure Service Bus. Not only that, but I’m a big fan of the .NET API which really feels oh-so-right and makes allowances for multiple patterns for synchronous vs. asynchronous code.

That being said, a big pain point with Service Bus has been using the Access Control Service for fine-grained security – which really can be the opposite of intuitive – especially when the concept of an identified user isn’t really needed or important to your integration scenario.

Thankfully, ACS isn’t the only security model that Service Bus supports. We actually can also use Shared Access Signatures for authentication. SAS authentication allows you to generate a key for a specific fine-grained entity within Service Bus for which you want to authorize access (e.g., Listen access for a given Queue), and then from that key you can generate a signed token that expires after a period of time. This token need not be borne by a specific user, it need only be presented to be accepted.

While all of that could be filed under mildly interesting, the important thing to note is that unless you have BizTalk Server 2013 R2 installed, you will be limited to using the ACS model for Service Bus authentication and authorization.

SAS-SY SB-Messaging Adapter

Thankfully, after upgrading to BizTalk Server 2013 R2, if you visit the Authentication tab of the SB-Messaging Transport Properties dialog, you will find the following option available:

image

Knowing that you can use Shared Access Signatures is one thing, being able to put that into practice is another. If you haven’t used SAS authentication/authorization before, you’re in for a treat.

Configuring SAS Authentication for Service Bus Entities

If you head over to the Microsoft Azure Management Portal, create some Service Bus entities, and then head to the Configure tab for the same, you will find a section titled Shared Access Policies

image

This section allows you to define access policies (e.g., read-only access, or Listen permission for the queue shown in the screenshot), and then generate signing keys that can be used to generate access tokens granting that level of permission.

image

It’s nice to know that this can all be done through the web UI if needed, but nothing here seems to relate back to the first property that you may have noticed when examining the settings for the SB-Messaging adapter (i.e., the Shared Access Key Name property). In reality, it’s asking for what the UI calls the Policy Name.

So what would the adapter configuration look like for this Shared Access Policy?

image

Putting it to the Test

So let’s put the updated adapter to the test and see what we get out the other end. First, let’s whip up a quick console app that will give us a message that is simply the string “Hello World”

image

Yes, the purists will note that I did not use a MesagingFactory. Why? Because this is not going to be a long-lived client, and it felt nicer to type. However, given a real world example, MessagingFactory will usually be the correct choice.

So let’s run down what I have while that message sits in the queue. I have a one-way Receive Port with a single Receive Location. This Receive Location uses the SB-Messaging adapter pointed at myqueue and using SAS for authentication (per the settings in the screenshot above). I have a Send Port subscribing to all messages from this Receive Port. This Send Port is using the FILE adapter, because I’m writing this late at night and am running out of creativity.

With everything in place, you will see this glorious sight…

image

And opening the file reveals…

image

Am I impressed that this traveled through a Service Bus Queue to arrive on my file system? No. I’m just happy that it authenticated using SAS token along the way, and I didn’t have to touch ACS at all during the process.

One hope that I have for this new functionality, is that it will see people moving beyond using owner for everything. Even though it’s something that I would find myself doing for demonstration purposes, it is also something that made me cringe to see in real life. It’s a lazy and even dangerous thing to do in some cases.

Just a Taste

This is really just a small flavor of what’s to come. There are some pretty big changes that aren’t getting a lot of fanfare at the moment, but I hope that will change as more and more people upgrade and discover what is waiting for them under the covers.

Until next week, take care!

Publishing a WCF Service via Service Bus Relay in BizTalk Server 2013 (Part 2)

This post is the eighth in a weekly series intended to briefly spotlight those things that you need to know about new features in BizTalk Server 2013. It is also the second part of a two-part series on the Service Bus Relay features in BizTalk Server 2013 (Click here for part 1).

Last week, we examined how BizTalk Server 2013 (and even 2010 in that case) can use the Windows Azure Service Bus Relay to make a service endpoint available in the cloud without otherwise directly exposing the machine, and while being able to take advantage of the claims-based authentication and authorization framework provided by the Access Control Service. This was done in such a way that the service was hosted in IIS rather than within an In Process BizTalk Host Instance. This week, we are going to look at the other side – a service endpoint that is ultimately hosted in IIS.

In this case, we have been provided with a few new adapters to help us along the way (although, if we were really adventurous, WCF-Custom could have still done the trick for us in the past). These two adapters are the WCF-BasicHttpRelay adapter and the WCF-NetTcpRelay adapter:

ServiceBusRelayInProc

Last week, we published a NetTcpRelay endpoint that required some additional work on the client to authenticate with the service. This week, we’re going to go to the other extreme, and publish a WCF-BasicHttpRelay endpoint that will be more or less transparent to the client.

Starting the Game of Telephone

We’re going to start out with the BizTalk version of Hello World, a simple (technically unnecessary, but convenient for illustrative purposes) orchestration. This orchestration receives a simple single node XML message (the single node containing the text “Hello”), and returns a simple single node XML message (containing the word “World”) as the response:

HelloOrch

The Setup

With this, we’re going to go through the typical process of creating an in-process receive location, and then walk through the BizTalk WCF Service Publishing wizard to publish the service. Then we’re going to consume the service as it sits. Once that is complete, we will change up the end point so that it is listening over a Service Bus Relay, and see that the client can still call the service.

We’ll start with the Receive Location. In order to keep this in-process, we’re actually using the WCF-Custom adapter (since WCF-BasicHttp will force us to an Isolated Host):

BasicHttpEP

Now that we have an endpoint to receive messages, we need to create an endpoint that informs clients what messages we’re even interested in – our MEX endpoint. For that, we will go back to the BizTalk WCF Service Publishing wizard, and select the second option – the path less traveled:

MEXPublish_Step1

We will walk through this wizard, ignoring everything that mentions Service Bus publishing for now. Ultimately, what we will end up with is a local endpoint that is exposing the WSDL necessary to consume the service. We can use the svcutil.exe utility to download and share this metadata offline, or even generate proxy classes for re-use later if we’d like. In this case, we will be consuming it in place, and generating a client that can be used to call the service no matter where it lives.

Since we started with an orchestration, and that will define the shape of the incoming/outgoing data, we will select the Publish BizTalk orchestrations as WCF service option. Do note the plural orchestrations in that sentence, I see way too many single operation services out there. Winking smile

MEXPublish_Step3

After browsing for and selecting an orchestration, you should be presented with a list of all public port types within the assembly containing the orchestration. If they Type Modifier property on the Port Type is not set to public, it will not be listed, and you will be sad:

selectport

After you click through a few more screens, you will be prompted for the location where you would like this metadata hosted. In this case, it will be hosted within IIS on the local machine:

MEXPublish_Location

A few more clicks through the wizard, and the MEX endpoint is created and hosted in IIS – though the service endpoint itself is still hosted in-process.

A visit to IIS manage demonstrates that the MEX endpoint is now:

http://localhost/IOServiceMEX/WcfService_QuickLearn_BizTalk_Process.svc?wsdl

Consuming the Service

So far, we haven’t really done anything earth shattering, so consuming the service is equally simple. In this case, I just fired up Visual Studio, used the Add Service Reference context menu option in Solution Explorer, and then provided the MEX endpoint to the wizard. With the (local, not relayed) Service Reference in place, I added the following code to my console app:

using System;

namespace QuickLearn.WCF.Client
{
    class Program
    {
        static void Main(string[] args)
        {
            var client = new IOServiceReference.WcfService_QuickLearn_BizTalk_ProcessClient("BasicHttpBinding_ITwoWayAsync",
                "http://localhost:81/Sample/IOService.svc");

            var response = client.ProcessMessage(new IOServiceReference.InOutRequest()
            {
                 Contents = "Hello"
            });

            Console.WriteLine(response.Contents);
            Console.ReadLine();
        }
    }
}

 

In the code above, the only moderately odd thing that I did was to use an overload of the client constructor that included an endpoint address. The reason for that was so I could more easily demonstrate the only change necessary as we switch to the WCF-BasicHttpRelay adapter.

To the Cloud

Since my client code is currently working and happy, I am ready to move this to the cloud. I am actually going to maintain my local endpoint (Receive Location) as well and simply add another Receive Location that will open the relay. The configuration of this one will be a little bit different, and certainly more involved.

NOTE: Before you continue, please be sure that you have read part 1.

I’m going to re-use my Azure Service Bus namespace from last week (though I have a new key for this week). I will start by creating a new receive location, selecting the WCF-BasicHttpRelay adapter, and then opening the adapter configuration. The first thing I am prompted for is the listen URI for the endpoint. This URI must incorporate my Service Bus Namespace, and should also add some identifier for the service after:

RelayEP_Config

The only other page of the configuration that I must visit is the Security page. This is where I get to determine if the client needs to provide some sort of authentication information in order to invoke the service, if the existence of the service is going to be discoverable by hitting the root of the namespace, and this is also where I set the authentication details for the access control service – so that BizTalk can prove to Azure that it can listen at that address and receive messages.

Since we don’t want to make changes to our client (other than the endpoint address), we will set the client authentication type to None. Then we need to click Edit under Access Control Service to bring up this dialog:

acs_credentials

In this dialog, we need to re-iterate our namespace (leaving the –sb suffix in-tact), and then then we also need to provide our super secret key which we can look up through the Azure Management Portal (see the previous post for details).

Once that is all in place, we’re set to Enable our Receive Location and then update the client to point to the relay.

Updating the Client

I mentioned before that we really only want to update the endpoint in the client. While this is true, the change of endpoint does (in this case) require a single simple change in configuration. The reason has nothing to do with the Service Bus Relay itself, but instead the change of scheme from http to https. An http address does not imply any security added in by the transport, it is just clear text over the wire. An https address, on the other hand, implies transport level security, as a result we need this small tweak to the otherwise auto-generated config file:

TransportSecurity

And then an update to our endpoint in code (everything else remains):

using System;

namespace QuickLearn.WCF.Client
{
    class Program
    {
        static void Main(string[] args)
        {
            var client = new IOServiceReference.WcfService_QuickLearn_BizTalk_ProcessClient("BasicHttpBinding_ITwoWayAsync",
                "https://uniquename.servicebus.windows.net/IOService/");

            var response = client.ProcessMessage(new IOServiceReference.InOutRequest()
            {
                 Contents = "Hello"
            });

            Console.WriteLine(response.Contents);
            Console.ReadLine();
        }
    }
}

This gives us the very happy result:

consoleapp_result

Clearly this is one of the most robust, secure, and complex ways to write the word “World” out to the console.

The Take-away

The take-away here for the last two posts is that BizTalk Server 2013 gives you some new options for exposing select endpoints over the internet without touching firewall configuration, without involving IIS (if you don’t want to involve it), and without exposing the whole box out to the internet. All the while giving you the ability to work with clients that are aware of the setup (in the case of the NetTcpRelay) or are blissfully ignorant of the magic behind the scenes (in the case of the BasicHttpRelay).

Windows Azure Service Bus Queues + BizTalk = Out of Body Experiences

This post is the third in a weekly series that will highlight things that you need to know about new features in BizTalk Server 2013. One of the major new features of BizTalk Server 2013, is the native ability to integrate with Windows Azure Service Bus through the new SB-Messaging adapter. This week we will be covering what can happen if you’re trying to integrate with applications that are loading up your Windows Azure Service Bus Queues with Brokered Messages made purely of properties.

Out of Context

Context properties are not a foreign concept to BizTalk developers. Indeed, BizTalk messages themselves are made up of both a message context and body parts. The same is true to some extent of Service Bus Brokered Messages – though the implementation differs greatly.

To deal with context properties of Service Bus Messages, the SB-Messaging adapter provides us the ability to promote these properties to the context of the BizTalk message generated upon receipt of a Brokered Message. The adapter configuration that controls this behavior looks like this:

sb-promote-props

One major difference between the properties of a Brokered Message and the context properties of a BizTalk message, is that BizTalk message context properties have a namespace to disambiguate properties that share the same name. Brokered messages have no such thing, and no concept of a property schema for that matter (which allows BizTalk to have an awareness of all possible properties that can exist in the context).

The adapter configuration setting pictured above is a bridge between those worlds that allows the Brokered Messages properties to be understood by BizTalk Server. Of course, you will have to define a property schema for those properties on the BizTalk side. Here’s an example of a really simple one that defines two custom properties:

propertyschema

All of this will function pretty perfectly – assuming that the message sent to the Queue has a body.

Out of Body

At last year’s TechEd NA conference in Orlando, one of the presenters (I believe it was Clemens Vasters) was discussing what Brokered Messages were, and how they were exchanged. In the talk, he mentioned that it was perfectly fine, and even preferable in some cases, to create a Brokered Message that was purely properties without any body content. Certainly this was a light-weight operation that relied only on HTTP headers, and required no overhead for serialization of body content. And at the time, I accepted this, and was happy.

As a result, a lot of my code interacting with the Service Bus involved snippets like this:

var client = Microsoft.ServiceBus.Messaging.QueueClient.CreateFromConnectionString(connString, queueName);
var message = new Microsoft.ServiceBus.Messaging.BrokeredMessage();
message.Properties.Add("CustomProperty", "Value");
message.Properties.Add("CustomProperty2", "Value2");
client.Send(message);

Notice here, we’re just passing around properties, and not dealing with a message body. That’s fine when I’m receiving the message in .NET code living somewhere else, but when moving into the BizTalk world, it became problematic. When attempting to receive such a message (even with the property schema deployed, and everything else otherwise happy), I would receive this (using the PassThruReceive pipeline):

errormsg

For those trying to use Google-fu to solve the same problem, the error message reads:

“There was a failure executing the receive pipeline: ‘Microsoft.BizTalk.DefaultPipelines.PassThruReceive, Microsoft.BizTalk.DefaultPipelines, Version=3.0.1.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35” Source: “Pipeline” Receive Port: “WontMatchWhatYouHaveAnyway” URI: “sb://namespacehere-ns.servicebus.windows.net/somequeuenamehere” Reason: The Messaging Engine encountered an error whlie reading the message stream.

If you came to this article because this error message appeared, and you’re not trying to receive an empty message from Azure Service Bus, just head right back to the Bing results page, and check the next result. Everyone else, stick with me.

Let’s take a step back for a second. BizTalk Server 2013 is designed for integration between systems. I typically should not be adjusting my behavior outside of BizTalk to work with BizTalk. So here I want to take the same approach.

What are our options here? Well clearly we have a failure in the pipeline while processing an entirely empty message body, which doesn’t allow us to get the point where we are processing a map. That removes our ability to use the excellent ContextAccessor functoid to pull the properties into the body of the message upon receipt.

Instead we are left with building a custom solution.

PropertyMessageDecoder Pipeline component

What is this custom solution I speak of? For now, I’m calling it the PropertyMessageDecoder pipeline component. It is a custom pipeline component that creates a message body based on the context properties of a BizTalk message. If you already have a message body, this pipeline component will not help you, since it blindly overwrites the body if one exists. Instead, this is a purpose-built pipeline component for the specific scenario addressed by this blog post.

I started development of this component by using Martijn Hoogendoorn’s BizTalk Server Pipeline Component Wizard. Unfortunately, it hasn’t yet been updated to work with BizTalk Server 2013, so I had to first create/upload a patch for BizTalk 2013.

Once I was able to create a custom pipeline component project, I created a simple pipeline component that took the input message, and dumped the context properties into the message bodies. I also included some configurable properties that allowed one to opt-out of properties from the BizTalk system properties namespace, or to filter out all properties except those from a specific namespace (especially helpful in this scenario).

Here’s the bulk of the Execute method for that component:

public Microsoft.BizTalk.Message.Interop.IBaseMessage Execute(Microsoft.BizTalk.Component.Interop.IPipelineContext pc, Microsoft.BizTalk.Message.Interop.IBaseMessage inmsg)
{
    Stream dataStream = new VirtualStream(VirtualStream.MemoryFlag.AutoOverFlowToDisk);
    pc.ResourceTracker.AddResource(dataStream);

    using (XmlWriter writer = XmlWriter.Create(dataStream))
    {
        // Start creating the message body
        writer.WriteStartDocument();
        writer.WriteStartElement("ns0", ROOT_NODE_NAME, TARGET_NAMESPACE);

        for (int i = 0; i < inmsg.Context.CountProperties; i++)
        {
            // Read in current property information
            string propName = null;
            string propNamespace = null;
            object propValue = inmsg.Context.ReadAt(i, out propName, out propNamespace);

            // Skip properties that we don't care about due to configuration (default is to allow all properties)
            if (ExcludeSystemProperties && propNamespace == SYSTEM_NAMESPACE) continue;
            if (!String.IsNullOrWhiteSpace(CustomPropertyNamespace)
                    && propNamespace != CustomPropertyNamespace) continue;

            // Create Property element
            writer.WriteStartElement(PROPERTY_NODE_NAME);

            // Create attributes on Property element
            writer.WriteStartAttribute(NAMESPACE_ATTRIBUTE);
            writer.WriteString(propNamespace);
            writer.WriteEndAttribute();

            writer.WriteStartAttribute(NAME_ATTRIBUTE);
            writer.WriteString(propName);
            writer.WriteEndAttribute();

            // Write value inside property element
            writer.WriteString(Convert.ToString(propValue));
            writer.WriteEndElement();
        }

        // Finish out the message
        writer.WriteEndElement();
        writer.WriteEndDocument();
        writer.Flush();

    }

    dataStream.Seek(0, SeekOrigin.Begin);
    var outmsg = Utility.CloneMessage(inmsg, pc);
    outmsg.BodyPart.Data = dataStream;

    return outmsg;
} 

The output will end up conforming to this schema:

propertymessageschema

And will look a little something like this:

<?xml version="1.0" encoding="utf-8"?>
    <ns0:PropertyMessage xmlns:ns0="http://schemas.quicklearn.com/PropertyMessage/2013/06/">
        <Property Namespace="http://schemas.example.org/" Name="CustomProperty1">Value1</Property>
        <Property Namespace="http://schemas.example.org/" Name="CustomProperty2">Value2</Property>
    </ns0:PropertyMessage>

When included in a pipeline alongside the XML disassembler component, we are finally able to take those pesky “empty” messages from Azure Service Bus and process them as XML documents. The message context properties are still retained in the context for routing, but our custom properties are also made available for use directly in maps (through the liberal use of Value Mapping (Flattening) and Logical functoids), and further processing.

To follow this through to the end, I created a quick proof-of-concept map:

samplemap

Essentially we’re looping through the properties from the context (hence using the Value mapping flattening functoid) and only mapping the value stored inside the Property node whenever the namespace and name of the property match those that we care about. Since the namespace for all of our custom properties is the same, we have a single Equals functoid that we’re sharing across all (two) properties.

After processing a message all the way through the bus, custom pipeline, and map, this should result in this beauty as opposed to a nasty error message:

sampleoutput

un-Borked brokered messages

Remember, that none of this stuff is necessary if you’re working with Brokered Messages that have a message body. You only need to go this deep when you’re dealing with Brokered Messages made purely of properties (which is a recommended practice when possible).

By bringing out-of-the-box first class support for Windows Azure Service Bus Queues, BizTalk Server 2013 continues to prove that with a solid and extensible architecture in place, any type of integration can be made possible.

If you would like to learn more about extending BizTalk Server 2013 to meet your integration challenges, check out one of our upcoming BizTalk 2013 Developer Deep Dive classes.

If you would like to access sample code for this blog post, you can find it on github.

BizTalk Server vNext TAP Program

We are pleased to announce to our customers and partners
that we will once again be participating in the Technology Adoption Program
(TAP) for the next version of BizTalk Server. Microsoft has yet to announce a
final name for the product, it is presently being referred to as BizTalk Server
2010 R2 or simply v-next, however they have indicated that the release date
will be about six months after the release of Windows 8 Server. I recently
posted another blog that talks about the future of BizTalk, check it out.

The TAP program assists Microsoft in soliciting feedback
from customers and partners regarding yet to be released products. As
participants in the TAP program we will be under strict non-disclosure
requirements so we won’t be able to say too much about the product until
Microsoft does, but rest assured that we will providing all the feedback that
we can to get all the cool features into the product that we have been looking
for.

 

 

QuickLearn has previously participated in the TAP program
for other versions of BizTalk and for Windows Azure. We will be prepared to
offer training on whatever new features are coming down the line as soon as the
product is fully baked!