Tuesday, December 14, 2010

Adventures with BizTalk: HTTP "GET" Part 4: Custom HTTP Adapter

[Note: This post is based upon an old blog post that I'm migrating for reference purposes, so some of the content might be a bit out of date. Still, hopefully it might help someone sometime...]

So, if we can't use the HTTP Send Adapter to retrieve files from a dynamically-obtained URL, can we override the behaviour in a custom HTTP Adapter?

Hmm... Well, first of all, writing a custom Adapter for BizTalk isn't something that I'd normally embark upon lightly. After all, you have the BizTalk OOTB Adapters and 3rd party Adapters, why write your own that you then have to test, support and maintain, particularly just to retrieve a file?!

So, in this case, I wasn't REALLY considering writing a custom HTTP Adapter... What I did briefly consider though was whether we could leverage the existing BizTalk HTTP Send Adapter, and just override its use of the "POST" HTTP verb.

The plan was coming together nicely, I'd identified the piece of code in the BizTalk HTTP Send Adapter that hardcodes the HTTP verb... It's in Microsoft.BizTalk.HttpTransport.dll, in the HttpAsyncTransmitter class's SetRequestProperties method:

request.Method = "POST"

So temptingly close, yet so far. Alas, the majority of the classes in Microsoft.BizTalk.HttpTransport.dll are sealed, and HttpAsyncTransmitter is one of them... So I couldn't inherit from it and override this one line!

I was left with the alternative of potentially disassembling the entire DLL and using it as the basis for writing my custom Adapter, or just going it alone...

Although I was already strongly leaning away from the custom Adapter approach, this really hammered the final nail in place...

So, it seems as though we've exhausted the alternatives for leveraging the OOTB BizTalk HTTP Adapters...

[UPDATE]

Another alternative to my rather brutal efforts to extend the OOTB BizTalk Adapter is to use the HTTP Adapter sample in the BizTalk SDK, also known as the "HTTP.NET" Adapter.

This sample is a rather rudimentary custom .NET implementation of an HTTP Adapter, so it's nowhere near as fully-featured as the OOTB HTTP Adapter, but it's a good start. And the good part is we have the full source code, so can change it to suit our needs.

The HTTP.NET Adapter implementation already supports specifying the HTTP verb as part of its configuration [why this hasn't been worked back into the OOTB HTTP Adapter I'm not sure].

In our case, the only change required came down to modifying the SendHttpRequest method in the HttpTransmitterEndpoint class (excerpt below):

private WebResponse SendHttpRequest(IBaseMessage msg, HttpAdapterProperties config)
{
  // ...
  // Original code...
  // Note: both the body part and the stream maybe null, so we
  // need to check for them
  string charset = string.Empty;
  IBaseMessagePart bodyPart = msg.BodyPart;
  Stream btsStream;

  if (request.Method != "GET")
  {
    // Original code...
    if (null != bodyPart && (null != (btsStream = bodyPart.GetOriginalDataStream())))
    {
      // ...
    }
  }

  return request.GetResponse();
}


The only change we've made is to wrap the code that deals with the request message body inside a check against the HTTP method (verb). If it's GET, we don't want to pass a request message body. Otherwise, we play on as per the original code.

This change could certainly be tidied up: It could be possible that there are some instances where we DO want to pass a body in the request message even when we're using GET, so you could introduce another configuration property such as "SuppressMessageBody" to cater for this.

The end-result however was that this change allowed us to achieve the desired result.

Looking back at it, I'm still not sure where my preference would be: I really do prefer to use OOTB Adapters where possible, not write and maintain my own, particularly when the custom version is a lot less feature-rich than the OOTB Adapter. However, I do kind of like using Adapters over custom .NET code... Now if the OOTB HTTP Adapter just included configuration properties for HTTP verb and message body suppression... Isn't this kind of requirement likely to become more prevalent with the increased focus on using REST?

When is String.Empty != String.Empty?

[Note: This post is based upon an old blog post that I'm migrating for reference purposes, so some of the content might be a bit out of date. Still, hopefully it might help someone sometime...]
Have just spent the last few days at a client site investigating a particularly obscure issue...

We've been working with a third-party API developed in .NET 1.1. A while ago, some ASMX services were built on top of the API, in VS2003 / ASP.NET 1.1. We wanted to upgrade these to VS2008, and eventually to WCF instead of ASMX.

So, I ran the upgrade on my development machine, everything upgraded fine, the services were functional, and I could successfully call the .NET 1.1 API. I deployed the services to an integration web server, and they stopped working with an obscure custom API error. I then spent a day reverse engineering the relevant bits of the API out into plain code using Reflector and the FileGenerator add-in (once again, yay for Reflector!), only to discover that the issue was with the following lines of code:

Private _OpenTags As String = ""
'...
If Not _OpenTags Is String.Empty Then
  '...

When the value of _OpenTags was "", the If test was returning False on my development machine (expected), True on the integration web server (unexpected).

OK... so after investigating potential differences in Framework configuration (eg, does Culture make a difference to how empty strings are evaluated?), and OS (development = WinXP, integration = Win2k3), I abandoned those paths and investigated the Framework versions on each machine.

Development machine had .NET 1.1, .NET 2.0 SP2, .NET 3.0 SP2, and .NET 3.5 SP1.

Integration machine had .NET 1.1, .NET 2.0 SP1, .NET 3.0 SP1, .NET 3.5 (no SP1).

OK... so, check out the list of fixes in .NET 3.5 SP1 (which also happens to include .NET 2.0 SP2 and .NET 3.0 SP2). Nothing there...

So, I built a test Win2k3 server, installed the same framework versions as the integration server, and ran my test... Sure enough, _OpenTags wasn't "equal to" (at least in a reference sense...) String.Empty, even when it was set to "". Installed .NET 3.5 SP1, and re-ran the test... Success: _OpenTags was String.Empty when set to "".

Eventually, to justify my desire to install .NET 3.5 SP1 on the integration server, I found the following in the .NET Framework 3.5 documentation:

http://msdn.microsoft.com/en-us/library/system.string.intern.aspx

The relevant part is the "Version Considerations" section, particularly the following statement:

In the .NET Framework version 3.5 Service Pack 1, the Intern method reverts to its behavior in the .NET Framework version 1.0 and .NET Framework version 1.1 with regard to interning the empty string.

... (example)

In the .NET Framework 1.0, .NET Framework 1.1, and .NET Framework 3.5 SP1, str1 and str2 are equal. In the .NET Framework version 2.0 Service Pack 1 and .NET Framework version 3.0, str1 and str2 are not equal.

Hmm... that would explain it... Thanks Microsoft. I haven't been able to find this listed in the "Breaking Changes" for .NET 2.0, maybe because it's so obscure, or just slipped their minds...

Note that it also depends on the way that you compare your string to String.Empty... In my case, as I was working with a third-party API, I didn't have any control over that...

Thanks for listening...

Thursday, December 9, 2010

CloudApp Part 1: Business Purpose & Architecture

A week or 2 ago I mentioned building my first app in the cloud and that I'd be blogging about some of the challenges this had posed. Well, I thought a good start (like in any project) would be to provide an overview of the business purpose the app was trying to fulfil, and the architecture we were proposing to employ.

Purpose

Although we started with a very lofty goal of providing a resource management application to help manage the resources within Chamonix, we decided that our first version should focus on capturing information about our people - things like basic resource profile information (name, office etc), as well as more interesting things like skills and education. This was a basic need for our sales guys to be able to know who had experience in what, whether it be technical, business, or industry oriented. Our plans were to build this basic data capture capability, and over time to flesh this out with further capabilities such as resource planning (potentially based on skills and availability).

Obviously a very important secondary goal for this exercise was to explore building apps in the cloud and get a better grip on some of the potential challenges - as well as some of the opportunities this presents. One of those opportunities (that we're still in the process of evolving) was the ability to allow users to sign in to our app using federated identity - eg Google, Facebook, or Windows Live. And of course while we were at it, if possible, to pull in information from their profile, such as their profile picture. As you'll see as we go along, this is getting increasingly easier, but also raises its own considerations.

Architecture

So, this is the high-level technology architecture we're using:
  • Data Store: SQL Server -> SQL Azure + Azure Storage
  • Data Access: Entity Framework 4.0
  • Business: WCF RIA Services
  • Presentation: ASP.NET 4.0
  • Authentication + Authorisation: Forms -> Azure Access Control Service
  • Hosting: IIS -> Azure Web Role
There are a few things to note here. Firstly, I've indicated roughly where we've made changes to the app in order to deploy it to the cloud. Secondly... Entity Framework 4.0 + WCF RIA Services + ASP.NET isn't such a common combination, but I wanted to try it out. You'll see my observations on the results in a future post...

Next time I'll get into some of the fun of building the parts of this app.

Monday, December 6, 2010

Adventures with BizTalk: HTTP "GET" Part 3: BizTalk HTTP Send Adapter

[Note: This post is based upon an old blog post that I'm migrating for reference purposes, so some of the content might be a bit out of date. Still, hopefully it might help someone sometime...]

In my last post on this subject, I investigated the use of the BizTalk HTTP Receive Adapter for retrieving files from a dynamically-obtained URL, with limited success. Today I'll look at the HTTP Send Adapter as an alternative option...

So, where the HTTP Receive Adapter polls a remote HTTP location looking for files that match its configuration and then retrieves them into the BizTalk MessageBox, the HTTP Send Adapter is (typically) used to send a message to a remote HTTP location. Most often this would be initiated either through BizTalk messaging or orchestration engine, with the message sent being constructed by the relevant engine before being POSTed to the remote URL.

And here's the problem with the HTTP Send Adapter... Even though HTTP includes a number of verbs that a client can use to initiate a request to a remote HTTP location, the BizTalk Send Adapter only supports "POST" - and what we want in this scenario is "GET". So although you could quite reasonably expect that you could use the HTTP Send Adapter to send out a request for a file via HTTP "GET", and to receive the response (the file itself) back, that's not the way the Adapter works. In fact, alas, the HTTP verb ("GET", "POST" etc) is hardcoded inside the Adapter's .NET code to only send messages via "POST".

Hmmm, so if using the OOTB HTTP Send Adapter isn't going to work, might we not be able to override this behaviour in a custom Adapter? Well, that's a question for next time...

Adventures with BizTalk: HTTP "GET" Part 2: BizTalk HTTP Receive Adapter

[Note: This post is based upon an old blog post that I'm migrating for reference purposes, so some of the content might be a bit out of date. Still, hopefully it might help someone sometime...]

Last time I gave an overview of some potential different approaches to achieving a dynamic HTTP GET using BizTalk.

In this post I'll talk about the first one of those, using the BizTalk HTTP Receive Adapter.

As you probably know, BizTalk includes an HTTP Receive Adapter, which is designed for the scenario where you point it at an HTTP URL, and the Adapter polls that location looking for files that match some criteria on a periodic basis. The idea of this (similar to other Receive Adapters) is that when a file matches, BizTalk pulls that file in through the Receive Location configured with the Adapter (using HTTP GET), and it can then be used by the BizTalk messaging and/or orchestration engines.

There are two main problems with using this Adapter for our scenario.

Firstly, the intended use for the HTTP Receive Adapter is when you know in advance what the URL you're going to be polling is, and can configure it "statically". In our case, we don't know this URL until we're processing our "source" message in our orchestration instance. And although you can dynamically set transport properties for Send Ports, it's not possible to do the same for Receive Ports / Locations.

You could do something tricky like dynamically creating Receive Ports / Locations, and indeed that's something I've considered in the past (but with the FTP Adapter) when you have the scenario of files that you want to receive from a fixed location, but whose file name changes each day. You can certainly achieve this using the classes in the BizTalk Explorer Object Model... The problem though is that the Explorer OM is only supported in a 32-bit process, and ideally we don't want to constrain ourselves in that way.
The second problem with this approach is that the HTTP Receive Adapter is a Receive Adapter, so it's mainly of use when you want to use it to poll a location, find a file that matches, receive the file into BizTalk, and activate an orchestration instance based on the file having been received. In our case though, we're already in an orchestration instance... so how do we trigger the Receive Adapter to start polling for a file, and even once it's been received by BizTalk, how can it be receieved by our "source" orchestration instance and / or by another dedicated orchestration that is somehow tied to our "source" instance.

Again, there are certainly ways that you can achieve this:

(a) Using dynamically created Receive Ports / Locations, and having the "source" orchestration Listen to the BizTalk MessageBox for a message that matches the file you're waiting for;
(b) Having a separate orchestration that "dumbly" receives the file (asynchronously) and then sends a "trigger" message that the "source" orchestration Listens for;
(c) If the "source" orchestration really doesn't care whether the file is received or not before continuing, it can send a message to the MessageBox that triggers a separate orchestration that asynchronously retrieves the file and performs any processing required.

I guess the main take-away from these two problems is that neither of them are exactly "easy" to overcome, and can potentially make the solution much more complicated (and less scalable) than is ideal.

Surely there must be a better alternative?