Integrate 2014 Starting Up

Tomorrow morning, the Integrate 2014 conference revs up here in Redmond, WA with Microsoft presenting the vision for the next versions of BizTalk Server and Microsoft Azure BizTalk Services.

I’m getting pretty excited as I’m looking over the first day agenda to see what to expect from the talks and sessions like Business Process Management and Enterprise Application Integration placed in between talks about BizTalk Services and On-Premise BizTalk Server. I’m even more excited seeing the second day agenda and seeing talks like Business Rules Improvements, and especially Building Connectors and Activities.

What’s Keeping Me Busy

In anticipation of this event, I have been building out a hybrid cloud integration sample application that leverages some of the ideas laid out in my previous post regarding cloud integration patterns while also providing the content for the long overdue next piece in my new features series for BizTalk Server 2013 R2.

In the sample, I’m running requests through MABS for content-based routing based on an external SQL lookup, with the requests targeting either an on-premise SQL Server database or a BizTalk Server instance for further processing that can’t otherwise be done currently with BizTalk Services (namely Business Rules Evaluation and Process Orchestration).

I’m hoping that after the conference this week, I will be able to tear the sample apart and build out most (if not all) of the elements in the cloud.

Best Bridge Between Azure and On-Prem?

Along the way, I’ve been further playing with using Service Bus Topics/Subscriptions as the Message Box of the cloud. At the same time, it represents a nice bridge between BizTalk Services and On-Premise BizTalk Server.

Consider the itinerary below;

image

This was actually the first draft of prototyping out the application. What this represents is an application that is receiving requests from client applications that have been published to a service bus topic. As they are received, they are routed based on content and sent either to a database to be recorded (as an example of an on-premise storage target), or to BizTalk Server 2013 R2 for further processing through a Service Bus Relay.

Given the scenario, maybe a relay is appropriate – lower latency requirement, no requirement for durability (which we have already sacrificed by running it through an initial bridge).

However, maybe we want to take a more holistic approach, and assume that the cloud is giving us a free public endpoint and some quite powerful content-based routing, translation, and even publish-subscribe capability when we bring Azure Service Bus to the mix. Let’s further assume that we view these capabilities as simply items in our toolbox alongside everything that BizTalk Server 2013 R2 is already providing us.

Let’s make it more concrete. What if the on-premise processing ultimately ends up sending the message back to the TradesDb? Is there a potential waste in building out that portion of the process in both locations?

Service Bus is the New Hybrid Integration MessageBox

Let’s try this instead:

image

Here, instead of using a relay, we’re using a Service Bus Topic to represent the call out to BizTalk Server 2013 R2.

Why would we do  this? While it introduces slightly more latency (in theory – though I haven’t properly tested that theory), it becomes pure loosely coupled pub-sub. I’m going to go out on a limb here (and risk being called an architecture astronaut), and say that not only is that not a bad thing, but it might even be a good idea. By feeding a topic rather than directly submitting to BizTalk Server via Relay allows us to easily swap out the processing of the rules with any mechanism we want, at any time – even if it means that we will have to transform the message and further submit by a completely different transport. Maybe one day we will be able to replace this with a call to a Rules Engine capability within MABS (crossing my fingers here) if we see such a capability come.

Further, we have broken out the logging of trades to the database into it’s own separate miniature process alongside the rest. This one might be fed by messages generated by BizTalk Server 2013 R2 on-premise or the earlier processing in MABS – providing only a single implementation of the interface to manage and change.

Is It The Right Way™?

I don’t know. It feels kind of right. I can see the benefits there, but again, we are potentially making a sacrifice by introducing latency to pay for a loosely coupled architecture. Call me out on it in the comments if that makes you unhappy. Ultimately, this would have to simply become a consideration that is weighed in choosing or not choosing to do things this way.

What Could Make it Even Better?

Imagine a time when we could build-out a hybrid integration using a design surface that makes it seamless. One where I could quickly discover the rest of the story. Right now there’s quite a bit happening on-premise into which we have no visibility via the itinerary – and very limited visibility within the orchestration since most logic is in Business Rules and my maps happen just before/after port processing.

Tomorrow

Tomorrow I will be writing a follow-up blog with a re-cap of the first day of the Integrate 2014 conference. Additionally, I will be playing with this application in my mind and seeing where the things announced this week change the possibilities.

If you’re going to be there tomorrow, be sure to stop by the QuickLearn Training table to sign-up for a chance to win some fun prizes. You can also just stop by to talk about our classes, or about any of the ideas I’ve thrown out here – I welcome both the positive and negative/nitpicky feedback.

Also, make sure you’re following @QuickLearnTrain on Twitter. We’ll be announcing an event that you won’t want to miss sometime in the next few days.

See you at the conference!

– Nick Hauenstein

BizTalk Server 2013 R2 Pipeline Component Wizard

While working on the current upcoming installment in my BizTalk Server 2013 R2 New Features Series, I realized that I did not yet have a version of Martijn Hoogendoorn’s excellent BizTalk Server Pipeline Component Wizard that would work happily in Visual Studio 2013.

As a result, I filed this issue, and then made the necessary modifications to make it work in Visual Studio 2013 with BizTalk Server 2013 R2. It is now available for download here (updated 27-FEB-2015) if you want to give it a test run.

image

Unfortunately, that has consumed a significant portion of my evening, and thus the next installment in the blog series will be delayed. But fear not, there is some really cool stuff just around the corner.

Integrate 2014 Registration is Live

In other news, Integrate 2014 registration is now live, and is only 72 days away! It’s going to be an awesome start to a cool Redmond December as 300-400 BizTalk enthusiasts from around the globe converge on the Microsoft campus for a few days of announcements, learning, networking, and community.

I’m not going to lie, I’m pretty excited for this one! Especially seeing these session titles for the first day:

  • Understand the New Adapter Framework in BizTalk Services
  • Introducing the New Workflow Designer in BizTalk Services

Well, that’s all for now. I hope to see you there!

Decoding JSON Data Using the BizTalk Server 2013 R2 JSONDecode Pipeline Component

This is the fourth in a series of posts exploring What’s New in BizTalk Server 2013 R2. It is also the second in a series of three posts covering the enhancements to BizTalk Server’s support for RESTful services in the 2003 R2 release.

In my last post, I wrote about the support for JSON Schemas in BizTalk Server 2013 R2. I started out with a small project that included a schema generated from the Yahoo Finance API and a unit test to verify the schema model. I was going to put together a follow-up article last week, but spent the week traveling, in the hospital, and then recovering from being sick.

However, I am back and ready to tear apart the next installment that already hit the github repo a few days back.

Pipeline Support for JSON Documents

In BizTalk Server 2013 R2, Microsoft approached the problem of dealing with JSON content in a way fairly similar to the approach that we used in the previous version with custom components – performing the JSON conversion as an operation in the Decode stage of the pipeline, thus requiring the Disassemble stage to include an XMLDisassemble component for property promotion.

The official component Microsoft.BizTalk.Component.JsonDecoder takes in two properties Root Node and Root Node Namespace that help determine how the XML message will be created.

Finally, there isn’t a JSONReceive pipeline included in BizTalk Server 2013 R2 – only the pipeline component was included. In other words, in order to work with JSON, you will need a custom pipeline.

image

 

Creating a Pipeline for Receiving JSON Messages

Ultimately, I would like to create a pipeline that is going to be reusable so that I don’t have to create a new pipeline for each and every message that will be received. Since BizTalk message types are all about the target namespace and root node name, it’s not reasonable to set that value to be the same for every message – despite having different message bodies and content. As a result, it might be best to leave the value blank and only set it at design time.

This is also an interesting constraint, because if we are receiving this message not necessarily just as a service response, we might end up needing to create a fairly flexible schema (i.e., with a lot more choice groups) depending on the variety of inputs / responses that may be received – something that will not be explored within this blog post, but would be an excellent discussion to bring up during one of QuickLearn’s BizTalk Server Developer Deep Dive classes.

In order to make the pipeline behave in a way that will be consistent with typical BizTalk Server message processing, I decided to essentially take what we have in the XMLReceive pipeline and simply add a JsonDecoder in the Decode stage, with none of its properties set at design time.

image

Testing the JSONReceive Pipeline

In the same vein as my last post, I will be creating automated tests for the pipeline to verify its functionality. However, we cannot use the built-in support for testing pipelines in this case – because properties of the pipeline were left blank, and the TestablePipelineBase class does not support per instance configuration. Luckily, the Winterdom PipelineTesting library does support per instance configuration – and it has a nifty NuGet package as of June.

Unfortunately, the per-instance configuration is not really pretty. It requires an XML configuration file that resembles the guts of a bindings file in the section dedicated to the same purpose. In other words, it’s not as easy as setting properties on the class instance in code in any way. To get around that to some degree, and to be able to reuse the configuration file with different property values, I put together a template with tokens in place of the actual property values.

image

NOTE: If you’re copying this approach for some other pipeline components, the vt attribute is actually very important in ensuring your properties will be read correctly. See KB884981 for details.

From there, the per-instance configuration is a matter of XML manipulation and use of the ReceivePipelineWrapper class’ ApplyInstanceConfig method:

private void configureJSONReceivePipeline(ReceivePipelineWrapper pipeline, string rootNode, string namespaceUri)
{
    string configPath = Path.Combine(TestContext.DeploymentDirectory, "pipelineconfig.xml");

    var configDoc = XDocument.Load(configPath);

    configDoc.Descendants("RootNode").First().SetValue(rootNode);
    configDoc.Descendants("RootNodeNamespace").First().SetValue(namespaceUri);

    configDoc.Save(configPath);

    pipeline.ApplyInstanceConfig(configPath);
}

The final test code includes a validation of the output against the schema from last week’s post. As a result, we’re really dealing with an integration test here rather than a unit test, but it’s a test nonetheless.

[TestMethod]
[DeploymentItem(@"Messages\sample.json")]
[DeploymentItem(@"Configuration\pipelineconfig.xml")]
public void JSONReceive_JSONMessage_CorrectValidXMLReturned()
{

    string rootNode = "ServiceResponse";
    string namespaceUri = "http://schemas.finance.yahoo.com/API/2014/08/";

    string sourceDoc = Path.Combine(TestContext.DeploymentDirectory, "sample.json");
    string schemaPath = Path.Combine(TestContext.DeploymentDirectory, "ServiceResponse.xsd");
    string outputDoc = Path.Combine(TestContext.DeploymentDirectory, "JSONReceive.out");

    var pipeline = PipelineFactory.CreateReceivePipeline(typeof(JSONReceive));

    configureJSONReceivePipeline(pipeline, rootNode, namespaceUri);

    using (var inputStream = File.OpenRead(sourceDoc))
    {
        pipeline.AddDocSpec(typeof(ServiceResponse));
        var result = pipeline.Execute(MessageHelper.CreateFromStream(inputStream));

        Assert.IsTrue(result.Count > 0, "No messages returned from pipeline.");

        using (var outputFile = File.OpenWrite(outputDoc))
        {
            result[0].BodyPart.GetOriginalDataStream().CopyTo(outputFile);
            outputFile.Flush();
        }

    }

    ServiceResponse schema = new ServiceResponse();
    Assert.IsTrue(schema.ValidateInstance(outputDoc, Microsoft.BizTalk.TestTools.Schema.OutputInstanceType.XML),
        "Output message failed validation against the schema");

    Assert.AreEqual(XDocument.Load(outputDoc).Descendants("Bid").First().Value, "44.97", "Incorrect Bid amount in output file");

}

After giving it a run, it looks like we have a winner.

image

Coming Up in the Next Installment

In the next installment of this series, I will actually put to use what we have here, and build out a more complete integration that allows us to experience sending JSON messages as well, using the new JsonEncoder component.

Take care until then!

If you would like to access sample code for this blog post, you can find it on github.

JSON Schemas in BizTalk Server 2013 R2

This is the third in a series of posts exploring What’s New in BizTalk Server 2013 R2. It is also the first in a series of three posts covering the enhancements to BizTalk Server’s support for RESTful services in the 2003 R2 release.

In my blog post series covering the last release of BizTalk Server 2013, I ran a 5 post series covering the support for RESTful services, with one of those 5 discussing how one might deal with JSON data. That effort yielded three separate executable components:

  1. JSON to XML Converter for use with the WFX Schema Generation tool
  2. JSON Decoder Pipeline Component (JSON –> XML)
  3. JSON Encoder Pipeline Component (XML –> JSON)

It also yielded some good discussion of the same over on the connectedcircuits blog, wherein some glitches in my sample code were addressed – many thanks for that!

All of that having been said, similar components in one form or another are now available out of the box with BizTalk Server 2013 R2 – and I must say the integrated VS 2013 tooling blows away a 5 minute WinForms app. In this post we will begin an in-depth examination of this improved JSON support by first exploring the support for JSON Schemas within a BizTalk Server 2013 R2 project.

How Does BizTalk Server Understand My Messages?

All BizTalk Server message translation occurs at the intersection between 2 components: (1) A declarative XSD file that defines the model of a given message, with optional inline parsing/processing annotations, and (2) an executable pipeline component (usually within the disassemble stage of a receive pipeline or assemble stage of the send pipeline) that reads the XSD file and uses any inline annotations necessary to parse the source document.

This is the case for XML documents, X12, EDIFACT, Flat-file, etc… It only logically follows then that this model could be extended for JSON. In fact, that’s exactly what the BizTalk Server team has done.

JSON is an interesting beast however, as there already exists a schema format for specifying the shape of JSON data. BizTalk Server prefers working with XSD, and makes no exception for JSON. Surprisingly this XSD looks no different than any other XSD, and contains no special annotations to reflect the message being typically represented as JSON content.

What Does a JSON Schema Look Like?

Let’s consider this JSON snippet, which represents the output of the Yahoo! Finance API performing a stock quote for MSFT:

image

This is a pretty simple instance, and it is also an interesting case because it has a null property Ask, as well as a repeating record quote that does not actually repeat in this instance. I went ahead and saved this snippet to the worst place possible – my Desktop – as quote.json and then created a new Empty BizTalk Server Project in Microsoft Visual Studio 2003 (with the recently released Update 3).

From there I can generate a schema for this document by using the Add New Items… context-menu item for the project within Solution Explorer. From there, I can choose JSON Schema Wizard in the Add New Item dialog:

image

The wizard looks surprisingly like the Flat-file schema wizard, and it looks like quite a bit of that work might have been lifted and re-purposed for the JSON schema wizard. What’s nice about this wizard though, is that this is really the only page requiring input (the previous page is the obligatory Welcome screen) – so you won’t be walking through the input document while also recursively walking through the wizard.

image

Instead the wizard makes some core assumptions about what the schema should look like (much like the WFX schema generator). In the case of this instance, it’s not looking so great. Besides from essentially every single element being optional in the schema, the quote record was not set as having a maxOccurs of unbounded – though this should really be expected given that our input instance gave no indication of this. However, maybe you’re of the opinion that the wizard may have been written to infer that upon noticing it was a child of a record with a plural name – which might be an interesting option to see.

image

Next the Ask record included was typed as anyType instead of decimal – which again should be expected given that it was simply null in the input instance. However, maybe this could be an opportunity to add pages to the wizard asking for the proper type of any null items in the input instance.

Essentially, it may take some initial massaging to get everything in place and happy. After tweaking the minOccurs and maxOccurs, as well as types assigned to each node, I decided it would be a good time to ensure that my modifications would still yield a schema that would properly validate the input instance I provided to the wizard.

How do We Test These Schemas Or Validate Our JSON Instances?

Quite simply, you don’t. At least not using the typical Validate Instance option available in the Solution Explorer context-menu for the .xsd file. Instead this will require a little bit of work in custom-code.

Where am I writing that custom code? Well right now I’m on-site in Enterprise, Alabama teaching a class that involves a lot of automated testing. As a result, I’m in the mood for writing some unit tests for the schema – which also means updating the project properties so that the class generated for the schema derives from TestableSchemaBase and adds a method we can use to quickly validate an instance against the schema.

image

It also means adding a new test project to the solution with a reference to the following assemblies:

  • System.Xml
  • C:\Program Files (x86)\Microsoft Visual Studio 12.0\Common7\IDE\PublicAssemblies\Microsoft.BizTalk.TOM.dll
  • C:\Program Files (x86)\Microsoft Visual Studio 12.0\Common7\IDE\PublicAssemblies\Microsoft.BizTalk.TestTools.dll
  • C:\Program Files (x86)\Microsoft Visual Studio 12.0\Common7\IDE\PublicAssemblies\Microsoft.XLANGs.BaseTypes.dll
  • Newtonsoft.Json (via Nuget Package)

That’s not all the setup required unfortunately. I still have to add a new TestSettings file to the solution, ensure that deployment is enabled, that it is deploying the bolded Microsoft.BizTalk.TOM.dll assembly above, and that it is configured to run tests in a 32-bit hosts. From there I need to click TEST > Test Settings > Select Test Settings File, to select the added TestSettings file.

image

image

image

image

With all the references in place and the solution all setup, I’ll want to bring in the message instance(s) to validate. In order to ensure that the test has access to these items at runtime, I will add the applicable DeploymentItem attribute to each test case that requires one.

using Microsoft.VisualStudio.TestTools.UnitTesting;
using Newtonsoft.Json;
using System.IO;
using System.Xml;

namespace QuickLearn.Finance.Messaging.Test
{
    [TestClass]
    public class ServiceResponseTests
    {

        [TestMethod]
        [DeploymentItem(@"Messages\sample.json")]
        public void ServiceResponse_ValidInstanceSingleResultNullAsk_ValidationSucceeds()
        {

            // Arrange
            ServiceResponse target = new ServiceResponse();
            string rootNode = "ServiceResponse";
            string namespaceUri = "http://schemas.finance.yahoo.com/API/2014/08/";
            string sourceDoc = Path.Combine(TestContext.DeploymentDirectory, "sample.json");
            string sourceDocAsXml = Path.Combine(TestContext.DeploymentDirectory, "sample.json.xml");

            ConvertJsonToXml(sourceDoc, sourceDocAsXml, rootNode, namespaceUri);

            // Act
            bool validationResult = target.ValidateInstance(sourceDocAsXml, Microsoft.BizTalk.TestTools.Schema.OutputInstanceType.XML);

            // Assert
            Assert.IsTrue(validationResult, "Instance {0} failed validation against the schema.", sourceDoc);

        }

        public void ConvertJsonToXml(string inputFilePath, string outputFilePath,
            string rootNode = "Root", string namespaceUri = "http://tempuri.org", string namespacePrefix = "ns0")
        {
            var jsonString = File.ReadAllText(inputFilePath);
            var rawDoc = JsonConvert.DeserializeXmlNode(jsonString, rootNode, true);

            // Here we are ensuring that the custom namespace shows up on the root node
            // so that we have a nice clean message type on the request messages
            var xmlDoc = new XmlDocument();
            xmlDoc.AppendChild(xmlDoc.CreateElement(namespacePrefix, rawDoc.DocumentElement.LocalName, namespaceUri));
            xmlDoc.DocumentElement.InnerXml = rawDoc.DocumentElement.InnerXml;

            xmlDoc.Save(outputFilePath);
        }

        public TestContext TestContext { get; set; }
    }
}

What Exactly Am I Looking At Here?

Here in the code we’re converting our JSON instance first to XML using the Newtonsoft.Json library. Once it is in an XML format, it should (in theory at least) conform to the schema definition generated by the BizTalk JSON Schema Wizard. So from there, we take output XML, and feed it into the ValidateInstance method of the schema to perform validation.

The nice thing about doing it this way, is that you will not only get a copy of the file to use within the automated test itself, but you can also use the file generated within the test in concert with the Validate Input Instance option of the schema for performing quick manual verifications as well.

After updating the schema, it looks like it’s going to be in a usable state for consuming the service:

Screenshot of final code

Coming Up Next Week

Next week will be part 2 of the JSON series in which we will test and then use this schema in concert with the tools that BizTalk Server 2013 R2 provides for consuming JSON content.

If you would like to access sample code for this blog post, you can find it on github.

BizTalk Integration Summit 2013 Recap

The week before this last holiday week was quite a time to be in the BizTalk world. The BizTalk Integration Summit 2013 kicked off at the swanky Fairmont Olympic Hotel in Seattle, Washington bringing with it a defined release cadence and roadmap, new cloud-based product announcements, and even a peek into the plans for the upcoming on-premise version of BizTalk Server.

The First Peek

The first peek into what the Thursday-Friday event would hold is captured in this agenda from the first day. The first day contained a single track of content, where all attendees would be able to experience the same messaging and presentations. The second day provided two tracks for those who wanted to chose their own adventure.

agenda

Familiar Faces & New Faces

The man in the Red Polo himself – Scott Guthrie kicked off the event – and also took the opportunity to announce the general availability of Windows Azure BizTalk Services. To clarify, it’s real and released – no longer in preview – but also not a static unchanging thing (as we will see later).

scottgu

We also had a new face bringing the initial welcome – Mark Moritmore. He’s the man behind the event, having received the torch passed by Kent Brown (who is now working with NY-based integration powerhouse Nimbo). This guy is passionate about the product, and is a long-term thinker. He’s exactly what BizTalk needs as organizations continue to choose BizTalk Server and Windows Azure BizTalk Services for mission critical integration applications that are expected to have long lifespans.

mark

Updated WABS SDK

The announcement of the general availability of Windows Azure BizTalk Services left me itching to get the bits, and “play” with them for lack of a better term. However, all the turkey from Thanksgiving put me a week behind the times and I just had a chance to download/install the freshly released SDK last night. One of the changes I noticed right away (from the last time I had installed one of the preview SDKs), is that I wasn’t having to hunt for miscellaneous MSIs to install the pre-requisite components. The new streamlined setup just makes it happen (though you’d better have a SQL Server or SQL Server Express instance handy if you’re planning on using the BizTalk Adapter Service – it’s not going to do that for you).

auto_install_dependencies

One of the other things demoed at the summit was the BizTalk Service Explorer Visual Studio Extension. This is a nice little plug-in for the Server Explorer window that allows you to interact with BizTalk Services artifacts in really cool ways (e.g., submitting a test message without leaving Visual Studio). While the UI looks/feels clunky at times, it definitely gets the job done. After seeing it, this will be on my list of extensions to install by default.

biztalkservice_explorer

Misc Thoughts About WABS GA

Looking at WABS from the perspective of migrating existing on-premise solutions to the cloud, I’m not sure the story is really complete there (and that’s not some huge secret – WABS is still in active development, and upcoming changes will erase most of the concerns I’m about to mention).

First, as noted in an earlier post, I did have some expectations around the map upgrade tool that still weren’t really met with this release. I had at least hoped the limitations in the preview would not be seen in the final release, or that I would get happier results than I was seeing before out of the tool at the very least.

Another tool that ships with the SDK is a TPM migration tool (essentially to move parties/profiles/agreements to the cloud. While this is actually done pretty well, there is still some mismatch in capabilities (e.g., EDIFACT is available on-premise, but is still a future delivery for WABS) that will prevent moving all solutions.

I took a look at the X12 schemas while thinking through some integrations I’ve worked on in the past, and noticed an absence of the schemas used for HIPAA sub-document splitting. To be fair, I haven’t actually had an opportunity to test to see if that functionality is present (for the most part the schemas are 100% clones of the on-premise schemas, so maybe those extensions are implemented in the WABS version of the EDI Disassembler), but it’s not looking promising. So those integrations would potentially stay on-premise as well.

Another thing that makes me more than a little bit uncomfortable is the direct deployment from Visual Studio. I’m hoping we will see more solutions like Michael Stephenson’s AppFx.ServiceBus.Build that will make deployment a happier experience (and hopefully a fully automated one that doesn’t live in Visual Studio).

That’s not the end of the story though. I don’t intend to write this in such a way that it seems I’m saying that WABS is somehow unsuitable. Instead these are the places where there is still work to be done – and I’ll emphasize again – work is being done. EDIFACT support was confirmed as coming, along with a new rules engine (with new tooling), support for Windows Workflow Foundation workflows, AS2 support, and other fun extensibility points (e.g., BAM-like tracking, easy to build/lightweight adapters). So we’re getting a better B2B story, and also starting to see the seeds of a BPM story emerge within WABS.

Essentially WABS is doing great things already (for straight-forward X12 integrations built fresh for WABS), and will be doing even better things in the future.

BizTalk 2013 Updates Since Release

Another point of discussion was the present state of BizTalk Server 2013 on-premise, as well as a road map for the future. Since the release of BizTalk Server 2013, the BizTalk team has been hard at work, and shipping updates and new functionality (e.g., HIS 2013, which had been delayed due to close work between the product team and TAP customers to get everything happy and ready for release).

2013updates

BizTalk Futures (2013 R2 and Beyond)

Looking to the roadmap ahead, the team made a commitment to a release cadence (that in a way is a continuation of what we have already seen, but which hadn’t so much been publicly formalized). The release cadence going forward will see:

  • Major BizTalk Server release every other year
  • R2 BizTalk Server release every year there is no major release
  • Cumulative update package each quarter
  • WABS refresh each quarter

release_cadence

Looking at this slide, I feel like I’m looking at a description of the release cadence for Visual Studio. Which is nice, because in a way this provides some level of guarantee that I’m going to be able to develop BizTalk Server artifacts in vLatest of Visual Studio (give or take a few months in this case).

As part of the commitment to this release cadence, an announcement was made for BizTalk Server 2013 R2, which will bring:

  • Platform alignment
  • JSON Support (so much for all that work writing pipeline components!)
  • Proxy support for the SFTP adapter (as well as SSO support)
  • Authentication improvements for Service Bus
  • Healthcare Accelerator Improvements (slides to follow with specifics)

2013r2features2013r2features2

Exciting Road Ahead

When you get down to the bottom of it, it really is an exciting road ahead for BizTalk Server. I had a lot of fun this year at the BizTalk Integration Summit, and I’m hoping that this becomes an annual tradition and opportunity for the BizTalk community to come together in-person, and have a dialog with the team that’s building the product and service.

Before I close this post out, I want to give a shout out and thanks to everyone that stopped by our table and has found themselves sporting a new tablet cover with the QuickLearn Training logo (spread the love!), and a thanks to everyone who made it to our party at the Rock Bottom Brewery – lots of good times and good conversations were had there!

If you want to get some other perspectives on the summit, Saravana and Steef Jan (among others) beat me to the punch and have offered their thoughts here:

Well, it looks like I don’t have an excuse to sit in this restaurant any longer (one of the upper floors of a seemingly endless concourse at DEN). I’m going to go catch the rest of my flight out to Houston (completely different part of Houston than last time) to teach some TFS this week, and I’ll be back in short order. Until then, take care!

Top 5 Indicators of the BizTalk Server 2013 Community’s Vitality

This post is the twenty-fourth, and final, in a weekly series intended to briefly spotlight those things that you need to know about new features in BizTalk Server 2013.

For my final post in this series, I wanted to highlight something that is not part of the core on-premise product, or even a part of Windows Azure BizTalk Services. Honestly, it was a struggle trying to figure out what to close out this series on. I had debated ending with the coverage on WABS, the code-sample heavy posts on REST, or even an end-to-end build-out of a solution using the new SharePoint adapter. Instead, I am going to focus on something which really causes BizTalk Server 2013 stand out from the rest, and is born out of its lengthy history as Microsoft’s flagship integration platform – the BizTalk Server 2013 Community.

So without further ado, I would like to present, the Top 5 Indicators that the BizTalk Server 2013 Community is thriving now more than ever.

Expanded Community of Bloggers

It has been exciting over the last few years to see more people get involved with blogging about BizTalk Server. The ESB Toolkit, specifically, is something near and dear to my heart, and I’m seeing consistently great things related to the Toolkit coming from people like Tomasso Groenendijk, and Colin Meade on their respective blogs. It’s especially cool, though, to see blogs specifically dedicated to BizTalk Server Administration– as with Tord Nordahl’s blog.1 If you ever want to be inspired and feel like a small part of a much larger community, just read through either Richard Seroter’s 4 question series, or Steef-Jan Wiggers’ BizTalk Community Series.

New Community Sites

Lex Hegt has done some pretty interesting work lately for the BizTalk Server Community, but he’s not taking any credit in the process. Just last year he launched BizTalkEvents.com which aggregates event schedules from BizTalk Server user groups and training classes from all over the world. It also features a presentation archive organized by date and event, complete with slide deck downloads.

EDIT: Michael Stephenson also pointed out that another interesting initiative that has come about during this latest round of BizTalk Server offerings is the BizTalk Maturity Assessment, which helps organizations determine their level of maturity across a variety of different areas, and then use the results of the assessment to grow those areas in which they need work.

Robust Third-Party Products

houstonmissioncontrol Since BizTalk Server is often used to build mission critical systems, it has tended to attract fairly high quality third-party products. This generation of 3rd party products entering the BizTalk Server 2013 ecosystem is certainly raising the bar though for quality and innovation. One really only needs to look at the latest (currently in Beta) version of Kovai’s BizTalk 360 headed up by Microsoft Integration MVP Saravana Kumar. Fresh out of the trappings of Silverlight, and into the land of HTML5 goodness, BizTalk 360 is augmenting the BizTalk Server Administration Console with something that would make even Houston jealous.

The TechNet Wiki

Every time I go to the landing page for BizTalk Server articles on the TechNet Wiki, I wonder to myself what percentage of the overall wiki is comprised of articles about BizTalk Server. Countless community contributors and careful curating by contributors like Steef-Jan Wiggers, have grown this free-form body of knowledge into a treasure trove of information about all versions of BizTalk Server – including pointers to the edges of the BizTalk Server Community itself.

The BizTalk Summit

Finally, if you want to see the BizTalk Server community thrive, go to one of the BizTalk Summit events. Hosted by Microsoft, these are special events where the community can come together, meet each other, swap battle stories, and bond over the shared love of bringing systems together. Last year, I had the opportunity (along with Rob Callaway, John Callaway, Chap Phillips and Paul Pemberton of QuickLearn) to bring some hands on labs to life, showing both Salesforce integration and an early peak at the EDI capabilities of WABS. I’m hoping to be a part of bringing even more this year – especially now that we have a platform for easy setup of such labs in Azure.

Registration is open for the summit right now, so don’t miss it!

The End?

While this post does mark the end of my series on the new features in BizTalk Server 2013, I’m not going to pack up my mouse and keyboard and call it a year. We’ll keep this space up to date with interesting tidbits that come up along the way. Some of it will be new stuff, some will be written responses to frequent questions that come up in class – but hopefully all of it will be adding value to the BizTalk Server community as a whole.

Take care until next time!

– Nick Hauenstein

1 If you’re ever stuck at a conference with people you don’t know, go find Tord and he’ll talk to you as if you were his best friend and keep you entertained in the process).

Mapping Code Pairs in the WABS Mapper

This post is the twenty-third in a weekly series intended to briefly spotlight those things that you need to know about new features in BizTalk Server 2013.

Assuming that one of the major benefits of using Windows Azure BizTalk Services is being able to offload some of our EDI processing to the cloud, then a major use case for the new WABS mapper becomes EDI mapping. One of the fun tasks that comes out of that is dealing with code pairs (if you’re interested in the on-premise side of this problem, the best coverage to date is found here). In this post, I’m going to examine how one might approach this problem using the new tooling available in the WABS SDK.

Give Me an Example of Code Pairs

Maybe you’re reading this and thinking to yourself, “I don’t really care about EDI, but WABS looks interesting,” and you still want to be able to gain something from this discussion without getting buried in the esoterica that is EDI. If that’s the case for anyone out there, here’s an example of the problem I’m going to try to solve using the WABS mapper.

We have an input that is a repeating record of items where there is a qualifier that describes that type of item we’re looking at, and a value that provides the value of the item. Consider this schema:

image

Here we have (at some point in the schema) a list of phone numbers, identified by a Type field element (our qualifier) and the Number field element (our value). The Type field is defined as follows:

image

We have an output that is a flat listing of items where we have a set of fields representing a few of these different types of numbers:

image

So what we are trying to do is iterate through the list of numbers in the input, map over the value for the item with the qualifier Mobile for the CellNumber output, and grab the value of the Home qualified number and toss it into the HomeNumber output field. It’s something that looks trivial, and yet is annoying to do en masse using the BizTalk Mapper:

image

In a Land Without FLATTENING Value MAPPERS

So what does this look like when brought over to the land of WABS using .trfm style maps? Well, looking into the toolbox, we do have an item called Conditional Assignemnt, which is essentially a Value Mapper:

image

If we run with that (combined with some Logical Expression map operations, we end up with something like this:

image

Unfortunately, this isn’t a Value Mapper (Flattening) type map operation. As a result, this actually will only work if we are interested only in a single qualifier, and that qualifier happens to show up in the first code pair that appears in the document.

So what do we do now? Do we reach to XSLT and code it up by hand (after-all, it is super easy to pull off)? Let’s take a look at how the migration tool that comes with the SDK handles this scenario:

image

It adds two MapEach operations to the map, that are configured something like this:

image

As you can see, the MapEach was made to be conditional upon the Type field element being compared to a constant string input. However, this conditional check is repeated inside the MapEach operation within the chained Logical Expression and Conditional Assignment map operations. While the map technically works as generated by the tool, it looks horrible, and it really feels like there ought to be a better way.

Another Approach

If you don’t want to use the MapEach approach (which really wouldn’t be so bad if our two nodes weren’t hanging out at the root as siblings), then we can take advantage of the ability of the new mapper to generate lists of items in memory while working through the mapping (really this is kind of similar to the Table Looping and Extracting functoid).

In order to make that happen, we use the Create List map operation, and then at some point inside include the Add Item operation to add items to the list. The Create List operation is configured to include the names of the fields of each item in the list. It then includes the ForEach operation with a nested Add Items. The ForEach is pointed to by the parent repeating node of our code pairs. The Add Items operation is pointed to by each value we want to include for each source item:

image

Here’s what the configuration of the Create List map operation looks like:

image

Once we have the list created, we can select values out of the list using the Select Value operation (in fact we’ve used two of them):

image

Does it Work?

Technically, both approaches worked, and took this input, and generated the output shown below it:

image

image

I’m going to be honest here, it still feels comparatively clunky, and I’m probably going to be reaching for that XSLT override pretty frequently. That is, unless there’s yet another way that’s cleaner and more simple than those presented thus far (this is one of those posts that I hope to revisit in the future with a correction showing an even better way to approach the problem).

Next Week

Next week, I will be wrapping up this series and shifting focus back to day-to-day BizTalk Development and all the fun things that we’re actively working on there.

Until then, happy mapping with shiny new tools!

Building Blocks of Windows Azure BizTalk Services

This post is the twenty-second in a weekly series intended to briefly spotlight those things that you need to know about new features in BizTalk Server 2013.

Since the last BizTalk Summit, there has been a lot of coverage within the greater BizTalk Community of Windows Azure BizTalk Services – the Azure-backed offering that brings pieces of the core BizTalk Server functionality to the cloud. This is something that QuickLearn had early opportunity to get hands-on with, and also something for which we were able to create labs (even before public availability). At the same time, it’s something that we haven’t really written about until now.

This week, rather than going through specific how-to guidance for pieces of the offering, I instead want to take a step back and look at the offering as a whole, so that we all understand the pieces that are there (and can see the forest through the trees). If you want to follow along, go ahead and download the Windows Azure BizTalk Services Preview SDK.

Once installed, you’ll find that Visual Studio has been juiced up with two additional project templates found under the BizTalk Services heading:

image

The first project type BizTalk Service allows you to define Itineraries (not in the ESB sense, though they do display a complete message flow, much like ESB), and the configuration for Bridges (much like a BizTalk Pipeline).

The second project type BizTalk Service Artifacts allows you to define schemas and maps (.trfm files, though you can convert your existing .btm files with some limitations using a tool provided at the SDK download link). The mapper is something I’m not going to be discussing any further, and as a result I will instead recommend this excellent blog post on the subject by Glenn Colpaert, and then also the official documentation on the conversion process/limitations.

So we can create these Itineraries, configure these Bridges, and deal with Schemas and Maps, but how does this all fit together within the context of Windows Azure BizTalk Services? To understand the answer to that, we have to first address the three things that Windows Azure BizTalk Services is trying to do:

  • Rich Messaging Endpoints (Itineraries + Bridges + Maps)
  • BizTalk Adapter Service (Service Bus Relay + Local Helper Service + BIzTalk Adapter Pack)
  • Business to Business Messaging (EDI Schemas + All of the Above)

Rich Messaging Endpoints

image

I’m going to start with the weirdest of the bunch – Rich Messaging Endpoints. The developer experience of this is similar (to a point) to defining an ESB Toolkit itinerary, but only inasmuch as you’re seeing the entire message flow within a single diagram. Here you can define certain sources (FTP/SFTP) for messages, and have them routed through Bridges (similar to Pipelines), and then routed out to certain destinations (for which CBR can be used to route the messages to one or multiple destinations).

In order to specify the settings necessary to connect to those destinations, you often find yourself within a configuration file defining WCF related binding settings directly:

image

Double-clicking on a bridge within the itinerary will bring up a semi-clone of the Pipeline Designer with a fixed set of components in place (as well as a location to specify maps to execute directly as part of the pipeline):

image

In terms of integration patterns, the bridges here are giving us a subset of the VETRO pattern (namely the VETR part of it): Validate, Enrich, Transform and Route. Where is the Route part of the equation you might ask? Well, you’ll actually find it in the Properties window for the bridge, by clicking the button in the Route Ordering Table property:

image

Here, we can do context-based routing that helps determine the destination for each message coming from the destination (though here, I have only connected a single destination endpoint).

BizTalk Adapter Service

image

What happens if the list of destinations doesn’t suit me? What if I want to take information from an SFTP drop, transform it into something that I could use to generate a record within a table in SQL, and then directly insert that record into my on-premise system? In that case, I’ll find myself reaching to the BizTalk Adapter Service feature.

This one has a nice list of dependencies that need to be in place before you can even use it (e.g., Windows Server AppFabric, BizTalk Server Adapter Pack), but once you have those, it’s fairly straight-forward to setup.

What it’s really providing is a single WCF endpoint that is exposed over a Service Bus Relay endpoint (and thus accessible from anywhere in the world – even if hosted behind a fairly strict firewall). This single endpoint can be passed messages destined for any number of internal systems that you can setup through the Server Explorer interface within Visual Studio. The BizTalk Team Blog actually had a pretty decent article on the topic back in June that sadly generated 0 comments.

Essentially, this is allowing you to bring your LOB systems into the mix to play along with everything else already mentioned (assuming those LOB systems have WCF adapters included in the BizTalk Adapter Pack).

Business to Business Messaging

The final capability that WABS is bringing to the table is B2B messaging (i.e., EDI). We have the ability through a special portal (still in preview) within the Windows Azure Management interface to create Parties and Agreements a la BizTalk Server on-prem. In fact, the same exact schemas are used to define the X12 messages, so if you’ve already had to do some schema customizations on a per-partner basis for in-house integrations, those same changes can now be brought to the cloud.

Pulling it All Together

Is this replacing BizTalk Server on-premise as we know it? Not quite. We have a lot of the same pieces: Transport (Limited)/Translation (Limited)/Transformation and Context-based Routing (Content-based Routing when using the Enrich stage of the Bridge). We are missing more complete process orchestration with exception handling / transactions / compensation (Orchestrations). We are missing rule-driven injectable logic (Business Rule Engine), among other things (though I’m stopping the list here to avoid debate about similar (but not quite) functionality in Azure in general).

So what do we do with this? Use it to solve integration problems that use a hybrid between cloud-based and on-premise resources (e.g., http://msdn.microsoft.com/en-us/library/windowsazure/hh859742.aspx). Use it when an on-premise BizTalk installation would be overkill for the task, but it can be done happily using the tools available in WABS. Use it to take over the most taxing parts of your B2B processing, leaving BizTalk Server on-premise to do those things it’s great at.

Ultimately, it’s up to you. BizTalk Server 2013 and Windows Azure BizTalk Services together give you the power to decide how to break up your integration work, and how you approach the integration challenges that are thrown your way.

If you haven’t yet given the SDK a download yet, go do it. Though the service is still in preview, the SDK has been available for quite some time now, and great things are right around the corner – so keep your skills sharp.

That’s all for now!

BizTalk Server 2013: Eliminating the Dependency on SQL Server 2005 Notification Services

This post is the twenty-first in a weekly series intended to briefly spotlight those things that you need to know about new features in BizTalk Server 2013.

Over the last few years, I’ve had my fair share of times describing to people the installation/configuration process for BizTalk Server and more specifically the Business Activity Monitoring component. During those discussions, the oddball requirement for SQL Server 2005 Notification Services comes up and sparks a mild level of controversy: “Why would I need a component from an older version of SQL Server to use BAM Alerts?” Thankfully, with BizTalk Server 2013, I won’t be having that discussion as frequently.

It Looks Like You’re Writing a Letter, Do You Want Some Help?

BizTalk Server 2013 is now able to take advantage of SQL Server 2012’s Database Mail feature to send BAM Alerts. It’s pretty straight-forward to get all setup. You start by taking a trip over to SQL Server Management Studio, and choosing Configure Database Mail on the item of the same name under the Management folder in Object Explorer:

image

From there, you walk through a fairly straight-forward wizard to create a new profile and specify the SMTP server / credentials to use to send email:

image

While we specify the account here as bam@quicklearn.com – that’s ultimately NOT the address the alert will be delivered FROM (which instead will be bam@microsoft.com):

image

As a quick test (of both the settings entered throughout the remainder of the wizard, and the local SMTP server), I used the Send Test E-Mail feature to ensure all was happy with the setup (but this step isn’t 100% necessary):

image

And this email was the result:

image

What Does the BAM Side of Configuration Look Like?

Assuming you’ve already gone through the standard BizTalk Server Configuration wizard and enabled/configured all of the BAM related functionality, it becomes a matter of deploying the activity/views you’re interested in, and then enabling alerts for those views:

image

From there, the BAM Portal will expose the alert options within the UI. As an example, I created a quick BAM View that incorporated the concept of processing time – with the idea that if even a single item took too long to process, an alert would be triggered.

For an alert like this, the activity search page is a fine place to configure the parameters. Here we start with a query as if we are searching for an item where the processing time is greater than 5 seconds, but instead of executing the query directly, you can click Set Alert to use the query criteria as the basis for an alert:

image

After you click the button, it directs you to a page where you can define what the message looks like, and then after that is setup, to add subscribers for the alert. The subscribers can be either email boxes that will receive the notification, or a file share:

image

Prove That This Works

In order to follow through all the way (without SQL Server 2005 Notification Services), I quickly threw together some code to populate the activity with some data that should satisfy the alert condition, and cause the alert to fire:

image

After executing the code, I found this beauty sitting in the drop folder ready for delivery:

image

BAM!

So what are the limitations here (because there’s always a catch right)? Well, you can take this approach provided that you are using SQL Server 2012 along with BizTalk Server 2013. If you’re using SQL Server 2008 R2, then we still have to have that awkward discussion, where I link you over to the download page for the Feature Pack for SQL Server 2005 SP4 and then die a little inside.

BizTalk Server 2013 is trying to do things right, and make some of these stories cleaner, and more logical. There will be 3 more posts in this series to finish driving that home, and complete the grand tour of the new functionality. From there, we will look to development / course development battle stories, and also vNext functionality as information becomes available.

Until then, stay tuned!

BizTalk Server 2013 Ordered Delivery Improvements

This post is the twentieth in a weekly series intended to briefly spotlight those things that you need to know about new features in BizTalk Server 2013.

Almost buried in the middle of the new features list, you will find that for those in the healthcare industry, Microsoft has finally delivered on a longer-term ask in the form of improvements to the handling of ordered delivery in BizTalk Server 2013. The ask was for better performance when using the MLLP adapter, but in reality – anyone who relies on ordered delivery with more than a single Send Port can benefit and see anywhere from 300-500% better performance.

This is the glimpse into the improvement that we get from the What’s New article:

In previous BizTalk versions with scenarios where ordered delivery uses multiple endpoints, the slowest adapter type is the maximum speed. This behavior directly impacts HL7 solutions. When using the MLLP adapter, acknowledgements (ACK) are required. Delays can occur because an ACK must be received before the next message is sent.

In BizTalk Server 2013, the next message can be sent without waiting for an ACK from the previous message. When the ACK arrives, it is internally correlated/connected with its message.

What was the Problem?

If you spend some time snooping around the blogs on MSDN from those involved in BizTalk Server development, you are likely to come across this post (back from when BizTalk Server 2013 hadn’t yet been christened 2013).

The post explains that in previous versions of BizTalk Server, whenever a host had a set of messages queued up and ready for delivery, if the first one was destined for an ordered Send Port, then the Host would be blocked until delivery of the message was successful (regardless of available threads, or other non-ordered Send Ports).

So what is the fix?

In the 2013 version of BizTalk Server, instead of relying only on the Host Queue, and blocking all messages that follow an ordered message, each Send Port with ordered delivery enabled is dedicated its own thread within the Host Instance, and messages sent to those locations will not block any other Send Ports that also have messages ready and waiting.

How Do I Take Advantage of This New Functionality?

That’s the really beautiful part about this feature; it’s built into the Messaging Engine. There is nothing special that you have to configure to enable this behavior, no code that you have to write – it’s just there and better by default. The only thing to keep in mind, is that since each ordered send port is dedicated its own thread, you may want to go to the BizTalk Settings Dashboard, and ensure that the  Messaging Engine Threads is appropriately configured to accommodate those Send Ports along with any other work that the Host has been assigned.

Best BizTalk Server Version Yet

If you haven’t made the move to BizTalk Server 2013 yet, or at least explored the evaluation to see what it has to offer, I would strongly encourage you to do so. Of course, if you want some dedicated hands-on time and instruction at the same time, I highly recommend our completely overhauled BizTalk 2013 Administrator Immersion class.

That’s all for this week. I’m back in the saddle after spending some time on vacation, so I will be back again next week. Until then, take it easy!