Decoding JSON Data Using the BizTalk Server 2013 R2 JSONDecode Pipeline Component

This is the fourth in a series of posts exploring What’s New in BizTalk Server 2013 R2. It is also the second in a series of three posts covering the enhancements to BizTalk Server’s support for RESTful services in the 2003 R2 release.

In my last post, I wrote about the support for JSON Schemas in BizTalk Server 2013 R2. I started out with a small project that included a schema generated from the Yahoo Finance API and a unit test to verify the schema model. I was going to put together a follow-up article last week, but spent the week traveling, in the hospital, and then recovering from being sick.

However, I am back and ready to tear apart the next installment that already hit the github repo a few days back.

Pipeline Support for JSON Documents

In BizTalk Server 2013 R2, Microsoft approached the problem of dealing with JSON content in a way fairly similar to the approach that we used in the previous version with custom components – performing the JSON conversion as an operation in the Decode stage of the pipeline, thus requiring the Disassemble stage to include an XMLDisassemble component for property promotion.

The official component Microsoft.BizTalk.Component.JsonDecoder takes in two properties Root Node and Root Node Namespace that help determine how the XML message will be created.

Finally, there isn’t a JSONReceive pipeline included in BizTalk Server 2013 R2 – only the pipeline component was included. In other words, in order to work with JSON, you will need a custom pipeline.

image

 

Creating a Pipeline for Receiving JSON Messages

Ultimately, I would like to create a pipeline that is going to be reusable so that I don’t have to create a new pipeline for each and every message that will be received. Since BizTalk message types are all about the target namespace and root node name, it’s not reasonable to set that value to be the same for every message – despite having different message bodies and content. As a result, it might be best to leave the value blank and only set it at design time.

This is also an interesting constraint, because if we are receiving this message not necessarily just as a service response, we might end up needing to create a fairly flexible schema (i.e., with a lot more choice groups) depending on the variety of inputs / responses that may be received – something that will not be explored within this blog post, but would be an excellent discussion to bring up during one of QuickLearn’s BizTalk Server Developer Deep Dive classes.

In order to make the pipeline behave in a way that will be consistent with typical BizTalk Server message processing, I decided to essentially take what we have in the XMLReceive pipeline and simply add a JsonDecoder in the Decode stage, with none of its properties set at design time.

image

Testing the JSONReceive Pipeline

In the same vein as my last post, I will be creating automated tests for the pipeline to verify its functionality. However, we cannot use the built-in support for testing pipelines in this case – because properties of the pipeline were left blank, and the TestablePipelineBase class does not support per instance configuration. Luckily, the Winterdom PipelineTesting library does support per instance configuration – and it has a nifty NuGet package as of June.

Unfortunately, the per-instance configuration is not really pretty. It requires an XML configuration file that resembles the guts of a bindings file in the section dedicated to the same purpose. In other words, it’s not as easy as setting properties on the class instance in code in any way. To get around that to some degree, and to be able to reuse the configuration file with different property values, I put together a template with tokens in place of the actual property values.

image

NOTE: If you’re copying this approach for some other pipeline components, the vt attribute is actually very important in ensuring your properties will be read correctly. See KB884981 for details.

From there, the per-instance configuration is a matter of XML manipulation and use of the ReceivePipelineWrapper class’ ApplyInstanceConfig method:

private void configureJSONReceivePipeline(ReceivePipelineWrapper pipeline, string rootNode, string namespaceUri)
{
    string configPath = Path.Combine(TestContext.DeploymentDirectory, "pipelineconfig.xml");

    var configDoc = XDocument.Load(configPath);

    configDoc.Descendants("RootNode").First().SetValue(rootNode);
    configDoc.Descendants("RootNodeNamespace").First().SetValue(namespaceUri);

    configDoc.Save(configPath);

    pipeline.ApplyInstanceConfig(configPath);
}

The final test code includes a validation of the output against the schema from last week’s post. As a result, we’re really dealing with an integration test here rather than a unit test, but it’s a test nonetheless.

[TestMethod]
[DeploymentItem(@"Messages\sample.json")]
[DeploymentItem(@"Configuration\pipelineconfig.xml")]
public void JSONReceive_JSONMessage_CorrectValidXMLReturned()
{

    string rootNode = "ServiceResponse";
    string namespaceUri = "http://schemas.finance.yahoo.com/API/2014/08/";

    string sourceDoc = Path.Combine(TestContext.DeploymentDirectory, "sample.json");
    string schemaPath = Path.Combine(TestContext.DeploymentDirectory, "ServiceResponse.xsd");
    string outputDoc = Path.Combine(TestContext.DeploymentDirectory, "JSONReceive.out");

    var pipeline = PipelineFactory.CreateReceivePipeline(typeof(JSONReceive));

    configureJSONReceivePipeline(pipeline, rootNode, namespaceUri);

    using (var inputStream = File.OpenRead(sourceDoc))
    {
        pipeline.AddDocSpec(typeof(ServiceResponse));
        var result = pipeline.Execute(MessageHelper.CreateFromStream(inputStream));

        Assert.IsTrue(result.Count > 0, "No messages returned from pipeline.");

        using (var outputFile = File.OpenWrite(outputDoc))
        {
            result[0].BodyPart.GetOriginalDataStream().CopyTo(outputFile);
            outputFile.Flush();
        }

    }

    ServiceResponse schema = new ServiceResponse();
    Assert.IsTrue(schema.ValidateInstance(outputDoc, Microsoft.BizTalk.TestTools.Schema.OutputInstanceType.XML),
        "Output message failed validation against the schema");

    Assert.AreEqual(XDocument.Load(outputDoc).Descendants("Bid").First().Value, "44.97", "Incorrect Bid amount in output file");

}

After giving it a run, it looks like we have a winner.

image

Coming Up in the Next Installment

In the next installment of this series, I will actually put to use what we have here, and build out a more complete integration that allows us to experience sending JSON messages as well, using the new JsonEncoder component.

Take care until then!

If you would like to access sample code for this blog post, you can find it on github.

JSON Schemas in BizTalk Server 2013 R2

This is the third in a series of posts exploring What’s New in BizTalk Server 2013 R2. It is also the first in a series of three posts covering the enhancements to BizTalk Server’s support for RESTful services in the 2003 R2 release.

In my blog post series covering the last release of BizTalk Server 2013, I ran a 5 post series covering the support for RESTful services, with one of those 5 discussing how one might deal with JSON data. That effort yielded three separate executable components:

  1. JSON to XML Converter for use with the WFX Schema Generation tool
  2. JSON Decoder Pipeline Component (JSON –> XML)
  3. JSON Encoder Pipeline Component (XML –> JSON)

It also yielded some good discussion of the same over on the connectedcircuits blog, wherein some glitches in my sample code were addressed – many thanks for that!

All of that having been said, similar components in one form or another are now available out of the box with BizTalk Server 2013 R2 – and I must say the integrated VS 2013 tooling blows away a 5 minute WinForms app. In this post we will begin an in-depth examination of this improved JSON support by first exploring the support for JSON Schemas within a BizTalk Server 2013 R2 project.

How Does BizTalk Server Understand My Messages?

All BizTalk Server message translation occurs at the intersection between 2 components: (1) A declarative XSD file that defines the model of a given message, with optional inline parsing/processing annotations, and (2) an executable pipeline component (usually within the disassemble stage of a receive pipeline or assemble stage of the send pipeline) that reads the XSD file and uses any inline annotations necessary to parse the source document.

This is the case for XML documents, X12, EDIFACT, Flat-file, etc… It only logically follows then that this model could be extended for JSON. In fact, that’s exactly what the BizTalk Server team has done.

JSON is an interesting beast however, as there already exists a schema format for specifying the shape of JSON data. BizTalk Server prefers working with XSD, and makes no exception for JSON. Surprisingly this XSD looks no different than any other XSD, and contains no special annotations to reflect the message being typically represented as JSON content.

What Does a JSON Schema Look Like?

Let’s consider this JSON snippet, which represents the output of the Yahoo! Finance API performing a stock quote for MSFT:

image

This is a pretty simple instance, and it is also an interesting case because it has a null property Ask, as well as a repeating record quote that does not actually repeat in this instance. I went ahead and saved this snippet to the worst place possible – my Desktop – as quote.json and then created a new Empty BizTalk Server Project in Microsoft Visual Studio 2003 (with the recently released Update 3).

From there I can generate a schema for this document by using the Add New Items… context-menu item for the project within Solution Explorer. From there, I can choose JSON Schema Wizard in the Add New Item dialog:

image

The wizard looks surprisingly like the Flat-file schema wizard, and it looks like quite a bit of that work might have been lifted and re-purposed for the JSON schema wizard. What’s nice about this wizard though, is that this is really the only page requiring input (the previous page is the obligatory Welcome screen) – so you won’t be walking through the input document while also recursively walking through the wizard.

image

Instead the wizard makes some core assumptions about what the schema should look like (much like the WFX schema generator). In the case of this instance, it’s not looking so great. Besides from essentially every single element being optional in the schema, the quote record was not set as having a maxOccurs of unbounded – though this should really be expected given that our input instance gave no indication of this. However, maybe you’re of the opinion that the wizard may have been written to infer that upon noticing it was a child of a record with a plural name – which might be an interesting option to see.

image

Next the Ask record included was typed as anyType instead of decimal – which again should be expected given that it was simply null in the input instance. However, maybe this could be an opportunity to add pages to the wizard asking for the proper type of any null items in the input instance.

Essentially, it may take some initial massaging to get everything in place and happy. After tweaking the minOccurs and maxOccurs, as well as types assigned to each node, I decided it would be a good time to ensure that my modifications would still yield a schema that would properly validate the input instance I provided to the wizard.

How do We Test These Schemas Or Validate Our JSON Instances?

Quite simply, you don’t. At least not using the typical Validate Instance option available in the Solution Explorer context-menu for the .xsd file. Instead this will require a little bit of work in custom-code.

Where am I writing that custom code? Well right now I’m on-site in Enterprise, Alabama teaching a class that involves a lot of automated testing. As a result, I’m in the mood for writing some unit tests for the schema – which also means updating the project properties so that the class generated for the schema derives from TestableSchemaBase and adds a method we can use to quickly validate an instance against the schema.

image

It also means adding a new test project to the solution with a reference to the following assemblies:

  • System.Xml
  • C:\Program Files (x86)\Microsoft Visual Studio 12.0\Common7\IDE\PublicAssemblies\Microsoft.BizTalk.TOM.dll
  • C:\Program Files (x86)\Microsoft Visual Studio 12.0\Common7\IDE\PublicAssemblies\Microsoft.BizTalk.TestTools.dll
  • C:\Program Files (x86)\Microsoft Visual Studio 12.0\Common7\IDE\PublicAssemblies\Microsoft.XLANGs.BaseTypes.dll
  • Newtonsoft.Json (via Nuget Package)

That’s not all the setup required unfortunately. I still have to add a new TestSettings file to the solution, ensure that deployment is enabled, that it is deploying the bolded Microsoft.BizTalk.TOM.dll assembly above, and that it is configured to run tests in a 32-bit hosts. From there I need to click TEST > Test Settings > Select Test Settings File, to select the added TestSettings file.

image

image

image

image

With all the references in place and the solution all setup, I’ll want to bring in the message instance(s) to validate. In order to ensure that the test has access to these items at runtime, I will add the applicable DeploymentItem attribute to each test case that requires one.

using Microsoft.VisualStudio.TestTools.UnitTesting;
using Newtonsoft.Json;
using System.IO;
using System.Xml;

namespace QuickLearn.Finance.Messaging.Test
{
    [TestClass]
    public class ServiceResponseTests
    {

        [TestMethod]
        [DeploymentItem(@"Messages\sample.json")]
        public void ServiceResponse_ValidInstanceSingleResultNullAsk_ValidationSucceeds()
        {

            // Arrange
            ServiceResponse target = new ServiceResponse();
            string rootNode = "ServiceResponse";
            string namespaceUri = "http://schemas.finance.yahoo.com/API/2014/08/";
            string sourceDoc = Path.Combine(TestContext.DeploymentDirectory, "sample.json");
            string sourceDocAsXml = Path.Combine(TestContext.DeploymentDirectory, "sample.json.xml");

            ConvertJsonToXml(sourceDoc, sourceDocAsXml, rootNode, namespaceUri);

            // Act
            bool validationResult = target.ValidateInstance(sourceDocAsXml, Microsoft.BizTalk.TestTools.Schema.OutputInstanceType.XML);

            // Assert
            Assert.IsTrue(validationResult, "Instance {0} failed validation against the schema.", sourceDoc);

        }

        public void ConvertJsonToXml(string inputFilePath, string outputFilePath,
            string rootNode = "Root", string namespaceUri = "http://tempuri.org", string namespacePrefix = "ns0")
        {
            var jsonString = File.ReadAllText(inputFilePath);
            var rawDoc = JsonConvert.DeserializeXmlNode(jsonString, rootNode, true);

            // Here we are ensuring that the custom namespace shows up on the root node
            // so that we have a nice clean message type on the request messages
            var xmlDoc = new XmlDocument();
            xmlDoc.AppendChild(xmlDoc.CreateElement(namespacePrefix, rawDoc.DocumentElement.LocalName, namespaceUri));
            xmlDoc.DocumentElement.InnerXml = rawDoc.DocumentElement.InnerXml;

            xmlDoc.Save(outputFilePath);
        }

        public TestContext TestContext { get; set; }
    }
}

What Exactly Am I Looking At Here?

Here in the code we’re converting our JSON instance first to XML using the Newtonsoft.Json library. Once it is in an XML format, it should (in theory at least) conform to the schema definition generated by the BizTalk JSON Schema Wizard. So from there, we take output XML, and feed it into the ValidateInstance method of the schema to perform validation.

The nice thing about doing it this way, is that you will not only get a copy of the file to use within the automated test itself, but you can also use the file generated within the test in concert with the Validate Input Instance option of the schema for performing quick manual verifications as well.

After updating the schema, it looks like it’s going to be in a usable state for consuming the service:

Screenshot of final code

Coming Up Next Week

Next week will be part 2 of the JSON series in which we will test and then use this schema in concert with the tools that BizTalk Server 2013 R2 provides for consuming JSON content.

If you would like to access sample code for this blog post, you can find it on github.

Automated Deployment and Testing of BizTalk Server 2010 Applications

Back when BizTalk Server 2009 was released, there was a lot of buzz about the new integrated support for unit testing of schemas, maps, and pipelines, and cleaner support for automated builds driven by Team Foundation Server (TFS). These were definitely welcome additions that made it much easier to enable continuous integration with automated build and testing. However, there were some things missing. In this article, I’ll review the current state of automated build and testing capabilities that exists for BizTalk Server 2010 when using Visual Studio 2010 Team Foundation Server (TFS). I will first examine the items lacking out of the box, and then direct you to community projects that can improve the situation.

When it comes to the built-in unit testing support in BizTalk Server 2009 and 2010, there are still some areas lacking. For example, testing schemas with imports yielded erroneous results, as all of the necessary schemas were not used in validation. Testing pipelines cannot be accomplished purely in memory with the API exposed. There was not really any work done to enable integration and functional testing of orchestrations. Finally, the automated build process did not take into account the necessity for the bits that were compiled to be deployed somewhere for testing — possibly due to the lack of functional and integration testing support for orchestrations (i.e., there would be no reason to deploy bits that could not be tested anyway).

Thankfully, community efforts filled in most of these gaps for BizTalk Server 2009, and indeed in most cases these community projects preceded the implementation of automated build and testing support in BizTalk Server. Pipeline, pipeline component, and flat-file schema testing was made much more elegant with Tomas Restrepo’s PipelineTesting library. Functional and integration testing of orchestrations was provided for in BizUnit, and automated deployment in preparation for test could be enabled using the bLogical Custom Tasks.

You may have used one or all of these community tools when setting up an environment with BizTalk Server 2009 and Team Foundation Server 2008, and it is still possible to use most of the functionality offered by these in BizTalk Server 2010 and Team Foundation Server 2010. If you have not had any experience with enabling automated testing of BizTalk Server applications, you may benefit from our self-paced online training course covering BizTalk testing strategies.

PipelineTesting Library

The PipelineTesting library is a mature and stable library that has been well maintained and has always remained a step ahead of what was offered out of the box with each version of BizTalk Server. It not only provides a less-clunky API, but a fuller feature set. This is best shown by a short example (although this example does not even get into the depth of what is provided in the PipelineTesting library).

Here is some rough sample code that shows basic usage of the built-in BizTalk Pipeline Testing support:

[TestMethod]
public void TimeReportFFReceivePipeline_TimeReportFF_OutputValidates()
{
    // Built-in BizTalk Pipeline Testing Support
    // (CON: Must be enabled on pipeline project before compilation)

    // CON: Must create the pipeline ahead of time
    TimeReportFFReceivePipeline target = new TimeReportFFReceivePipeline();

    // PRO: Can provide a list of input documents without loading them explicitly
    // CON: Input documents must exist on disk
    StringCollection inputDocuments = new StringCollection();
    inputDocuments.Add(Path.Combine(TestContext.TestDeploymentDir, "TimeReport_3.txt"));

    StringCollection parts = new StringCollection();

    Dictionary schemas = new Dictionary();
    // CON: Must reference .xsd file
    schemas.Add("TestWorkshopApplication.Messaging.TimeReportFF", @"......MessagingTimeReportFF.xsd");

    target.TestPipeline(inputDocuments, parts, schemas);

    // CON: Must locate output messages as files
    DirectoryInfo testDir = new DirectoryInfo(TestContext.TestDeploymentDir);
    FileInfo[] outputList = testDir.GetFiles("*.out");

    Assert.IsTrue((outputList.Length > 0), "No outputs for pipeline found");

    foreach (FileInfo file in outputList)
    {

        /* Additional testing of message content */

    }
}

Now compare that with a code performing roughly the same test using the PipelineTesting library:

[TestMethod]
public void TimeReportFFInPipeline_TimeReportFF_OutputValidates()
{

    // PipelineTesting Library Pipeline Testing Support

    String sourceFile = Path.Combine(TestContext.TestDeploymentDir, "TimeReport_4.txt");

    // PRO: Can compose a pipeline at runtime
    FFDisassembler ffDasm = Disassembler.FlatFile().WithDocumentSpec(typeof(TimeReportFF));
    ReceivePipelineWrapper target = Pipelines.Receive().WithDisassembler(ffDasm);
    target.AddDocSpec(typeof(TimeReportFF));

    MessageCollection result;

    // CON: Have to manually load input document
    // PRO: Input document can exist only in memory
    using (var readStream = File.OpenRead(sourceFile))
    {
        IBaseMessage msg = MessageHelper.CreateFromStream(readStream);
        result = target.Execute(msg);
    }

    Assert.IsTrue(result.Count > 0, "No outputs found from flat file pipeline");

    // PRO: Raw IBaseMessage instances of outputs are available in memory
    foreach (IBaseMessage msg in result)
    {

        /* Additional testing of message content or context */
    }

}

It actually doesn’t take too much effort to get this library working with your BizTalk Server 2010 solutions, and in turn with Team Foundation Server 2010. You will have to upgrade the solution to the Visual Studio 2010 format, and re-reference the PipelineObjects library (exists under the Program FilesMicrosoft Visual Studio 10.0Common7IDEPublicAssemblies folder). If you don’t have NUnit installed, you can exclude the Test project from the build. Once you have it built, you’re good to start writing tests against your BizTalk Server 2010 artifacts. When automating the build and test with Team Foundation Server, you will need to remember to include the .dll somewhere within source control.

BizUnit 3.1

BizUnit is a framework for performing automated integration and functional testing of BizTalk solutions. It takes a black-box approach, and requires that all artifacts are built and deployed before testing commences. If you already perform manual end-to-end testing of your BizTalk applications, you could likely benefit greatly from BizUnit over time. It provides a huge library of pre-built test steps (e.g., start BizTalk host, copy a file, wait for a file, execute a database query) that have the net effect of saving you a lot of time. Test cases, which are combinations of test steps, exist in a declarative XML format that are processed by the BizUnit class which can be invoked from within your MSTest or NUnit test assemblies.

Getting BizUnit working for BizTalk Server 2010 is not necessarily a very easy process. Before you even begin, you must consider that there are two versions currently available. The most recent version is BizUnit 4.0 Beta, which completely changes the test case format (to use XAML), and looks to be the way of the future. However, it does not look like that version has been updated in quite some time, so it is unclear when a final release will be ready. The other version available is BizUnit 3.1, which is stable and has a fully developed library of test steps along with full documentation. For the sake of this blog posting, we will go forward with 3.1, and examine some of the issues you will encounter.

This is another instance in which you will have to upgrade the solution to the Visual Studio 2010 solution file format. From there, you will notice that the solution has a project for each category of test step (e.g., BizTalk steps, LoadGen steps, MQ Series steps, Outlook automation steps, etc…). You can exclude the project for any type of step that you will not be using. Then for each of the “Steps” projects, you will need to re-reference assemblies, so that the latest versions are referenced. You might notice at this point, that the BizTalkSteps project has a dependency on the PipelineTesting library already discussed. You will definitely want to include the version that you built against the BizTalk Server 2010 binaries here. This is another place where you will need to remember to include the .dll’s for both the BizUnit runtime and the steps that you will be using within source control, or simply install BizUnit on each build server if that makes more sense in your environment.

bLogical Custom Tasks

With BizTalk Server 2009, it became possible to automate builds of BizTalk applications without installing Visual Studio on the build server (though it is required to install the Project Build Component from the BizTalk installation media). It was also possible with BizTalk Server 2009 to automate tests and, with the help of community tools, automate integration and functional tests. However it was still not possible, using only out-of-the-box functionality, to include deployment as part of the build. The custom MSBuild Tasks for BizTalk by bLogical were developed to add the missing deployment step to the automated build process. Unfortunately, there are a few issues that still need to be overcome for these to work with Team Foundation Server 2010.

First, the easiest problem to deal with is in the GacHelper.cs class within the project:

namespace Blogical.Shared.Tools.BizTalkBuildTasks.Internal
{
	public class GacHelper
	{
		#region Private Fields
		const string ToolName = "bin\gacutil.exe";
		#endregion

This path gets concatenated with the path to the .NET Framework SDK to resolve the full path to the gacutil tool. Unfortunately, this will reference the old gacutil tool. For .NET 4 assemblies, the new tool path should be "bin\NETFX 4.0 Toolsgacutil.exe".

The second hurdle isn’t so easily solved. There is a dependency, mainly in the internal TeamBuildTask base class, on classes within the Microsoft.TeamFoundation.Build.Proxy namespace. This has been obsoleted, and no longer appears in the 2010 version of the assembly. This means that, at least for now, you will now have to provide your own solution for these small missing pieces in the automated build/deployment process. Post a comment if you have found an alternative solution, or keep watching this space for updates on this.

Summary

Using BizTalk Server 2010, you can still benefit from the automated build and testing capabilities released with BizTalk Server 2009, and most of the community solutions developed to enable continuous integration. There are a few steps necessary to get everything working and setup, and a few missing pieces that you will have to build yourself, but the end result is beautiful.

Also, bear in mind that Team Foundation Server is not just about source control and it’s not just about automated build or testing. It’s also about issue tracking, project management, and reporting. If you are paying for a license, make sure that your organization is taking full advantage of it — especially on your BizTalk Server integration solutions. Be sure to check our new collection of instructor-led training on Application Lifecycle Management using Visual Studio 2010 and Team Foundation Server 2010.