Howard Dierking

Vanilla Yet Composable HTTP Services With WCF 4 and Windsor

hdierking — Tue, 19 Jan 2010 07:30:00 GMT

Recently in my team, there’s been a lot of discussion about how we should go about implementing a service layer for various orthogonal behaviors. At the center of the conversation is the question of whether or not we should develop this layer using MVC or WCF. On first look, MVC paints a pretty compelling picture – particularly since the majority of our services will be consumed in a simple HTTP services fashion (Note: I’m intentionally avoiding the term RESTful service because these services will not necessarily be resource/noun focused – however, they will be plain old JSON or XML over HTTP – not something that requires a proxy in order to work with it). Specifically, here are the some of the things that makes MVC look appealing:

It’s really easy – just create a controller and action method
Friendly URLs ala core routing
IIS hosted
JSON formatting is as simple as returning the JSON action result from the action method
It’s easy to leverage an IoC container – simply create/plug in a custom controller factory

So that’s a pretty good list. However, even with all of those benefits, I kept coming back to a couple problems with using MVC as a general purpose service layer.

It tightly couples your service to a specific host (IIS)
It tightly couples your service to protocol (HTTP)
it tightly couples your service to an addressing scheme (URL)

Now I know that some of you are reading this and calling YAGNI on me – and that may be fair to some degree. However, for what will be publically facing services (which implies that there needs to be some degree of longevity), I’m hesitant to take a dependency on an underlying “service platform” (in quotes since MVC wasn’t really meant to be this) that takes so many dependencies on things that I consider to be an architectural style (remember, it wasn’t that long ago that SOAP was the last thing we would ever need – and DCOM/CORBA before that). And technically, I’m not really talking about building anything based on unknown future requirements, but on choosing a platform on which to take a dependency.

Philosophically, I’m a big fan of WCF because as one of its key characteristics, it decouples the programming model (the service definition) from both the networking stack that exposes and pipes data into the service and from the process that hosts and manages the service. However, the initial versions of WCF provided all of this philosophical goodness at a cost – and that cost was complexity – particularly in the area of the host and network stack configuration. And that complexity grew pretty significantly the further off the path you traveled.

That said, with each successive release of WCF, the level of complexity in configuration has been shrinking – to the point where as of WCF 4, the configuration piece has dwindled away to almost nothing.

At any rate, like I mentioned, I wanted to see whether I could build simple JSON over HTTP services in a way that leveraged all of WCF’s philosophical goodness but would give me the simplicity of development that I would find doing it in MVC. More specifically, here’s what I was testing for:

Standard WCF programming model
JSON by configuration
Use of IoC to resolve the service graph
Hosted in IIS
Friendly URLs

Jumping right in, WCF has a really straightforward programming model. It centers around the use of .NET interfaces decorated with a few attributes – these (along with some configuration around address and network stack) are exposed as WCF endpoints – and classes that implement one or more of those interfaces – these are WCF services and are the smallest units when it comes to hosting. The big idea here is that by using plain old .NET constructs, you can completely decouple your service logic and even contracts from all the details around hosting it and exposing it to the network. I started out in developing my service against this standard programming model.

[ServiceContract(Namespace = "urn:Spikes-WCF-JSONTests:IPersonService")]
public interface IPersonService
{
    [OperationContract]
    [WebGet(UriTemplate = "Find/{firstName}", BodyStyle = WebMessageBodyStyle.Bare, ResponseFormat = WebMessageFormat.Json)]
    Person FindPerson(string firstName);
}

As you can see, I’ve got a standard service contract with one operation. The WebGetAttribute allows me to specify a URI template with parameter placeholders. It’s worth noting here that this doesn’t give us complete MVC-ish friendly URLs – this is still relative to the service address, which if it’s hosted in IIS is that *.svc file. However, as I’ll show in a second, WCF 4 helps us remedy this little bit of trailing ugliness. Additionally, I’ve declared my message formatting in the WebGetAttribute class. I suspect that this can actually be externalized to config, but I haven’t figured out how to do this yet, and and attribute is still less invasive than returning a JsonActionResult in MVC – so for now, that’s what I’m going with.

The service itself is a pretty straightforward implementation of this interface – with one exception..

[AspNetCompatibilityRequirements(RequirementsMode = AspNetCompatibilityRequirementsMode.Allowed)]
[ServiceBehavior(InstanceContextMode = InstanceContextMode.PerCall)]
public class PersonService : IPersonService
{
    readonly IPersonRepository _personRepository;

    public PersonService(IPersonRepository personRepository) {
        _personRepository = personRepository;
    }

    public Person FindPerson(string firstName) {
        return _personRepository.GetPeople().FirstOrDefault(p => p.FirstName == firstName);
    }
}

Notice that I have added a parameter to my constructor which declares that my service has a dependency on a concrete implementation of IPersonRepository. One of the things that I love about MVC is how straightforward it is to create a custom ControllerFactory that can use an IoC container to resolve a controller’s entire dependency graph. As such, I wanted to make sure that I could do the same thing in WCF. Early on in my searching, I was very fortunate to come across a 2 part blog series on doing just this. I have adapted the source just a bit to use Windsor and to account for a thing or 2 in WCF 4, but the bulk of the code (and my sincerest thanks) go to Fabio Cozzolino.

In short, the ControllerFactory equivalent in WCF is called the instance provider, and it’s any class that implements IInstanceProvider. Here’s an instance provider that’s built to use a Windsor container:

class WindorInstanceProvider : IInstanceProvider
{
    readonly WindsorContainer _container;
    readonly Type _contractType;

    public WindorInstanceProvider(WindsorContainer container, Type contractType) {
        _container = container;
        _contractType = contractType;
    }

    public object GetInstance(InstanceContext instanceContext) {
        return GetInstance(instanceContext, null);
    }

    public object GetInstance(InstanceContext instanceContext, Message message) {
        return _container.Resolve(_contractType);
    }

    public void ReleaseInstance(InstanceContext instanceContext, object instance) {
        _container.Release(instance);
    }
}

I would elaborate, but there’s just not that much of interest going on here. The instance provider is constructed with the container itself and the service type. When asked, it turns around and asks the container to resolve the dependency graph and create an instance of the service type. What’s more interesting is how the instance provider gets plugged in so that it can do its thing.

In the same way that ASP.NET (and MVC by extension) has a request pipeline, WCF has a channel stack. However, unlike the case of ASP.NET, the WCF channel stack was meant to work across a myriad number of protocols and network topologies – as such, it’s a great deal more extensible – and that extensibility can sometimes bring with it some additional complexity. In this case, that complexity exists in the form of a couple extra points of indirection – the first of which is a service behavior.

public class WindsorServiceBehavior : IServiceBehavior
{
    readonly WindsorContainer _container;

    public WindsorServiceBehavior(WindsorContainer container) {
        _container = container;
    }

    public void Validate(ServiceDescription serviceDescription, ServiceHostBase serviceHostBase) {}

    public void AddBindingParameters(ServiceDescription serviceDescription, ServiceHostBase serviceHostBase,
                                     Collection<ServiceEndpoint> endpoints,
                                     BindingParameterCollection bindingParameters) {}

    public void ApplyDispatchBehavior(ServiceDescription serviceDescription, ServiceHostBase serviceHostBase) {
        foreach (ChannelDispatcher channelDispatcher in serviceHostBase.ChannelDispatchers)
            foreach (var endpointDispatcher in channelDispatcher.Endpoints) {
                if (endpointDispatcher.ContractName == "IMetadataExchange" || endpointDispatcher.ContractName == "IHttpGetHelpPageAndMetadataContract")
                    return;
                string contractName = endpointDispatcher.ContractName;
                ServiceEndpoint serviceEndpoint =
                    serviceDescription.Endpoints.FirstOrDefault(e => e.Contract.Name == contractName);
                endpointDispatcher.DispatchRuntime.InstanceProvider =
                    new WindorInstanceProvider(_container, serviceEndpoint.Contract.ContractType);
            }
    }
}

The purpose of my service behavior is simply to iterate through each endpoint in my service and set the endpoint’s instance provider to a new instance of my WindsorInstanceProvider, passing it an instance of my container (which I supply in the behavior’s constructor). I’m only doing this for custom endpoints – not for the metadata endpoint sometimes added somewhat transparently by a different behavior. It’s also worth pointing out that with WCF 4, I need to add a second type of metadata endpoint, IHttpGetHelpPageAndMetadataContract (I fully expect that we’ll see interface names that are complete sentences before long), to the exclusions list.

So you can now see where the instance provider comes from, but what about the container itself? And where is the behavior wired up. Enter the custom service host and custom service host factory First the factory:

public class WindsorServiceHostFactory : WebServiceHostFactory
{
    readonly WindsorContainer _container;

    public WindsorServiceHostFactory() {
        _container = new WindsorContainer(new XmlInterpreter(new ConfigResource("castle")));
    }

    public override ServiceHostBase CreateServiceHost(string constructorString, Uri[] baseAddresses) {
        //ctor string is the qualified serviceType name 
        var t = Type.GetType(constructorString);
        if (t == null)
            throw new ArgumentNullException("Could not resolve type '" + constructorString + "'");
        return CreateServiceHost(t, baseAddresses);
    }

    protected override ServiceHost CreateServiceHost(Type serviceType, Uri[] baseAddresses) {
        var serviceHost = new WindsorServiceHost(_container, serviceType, baseAddresses);
        return serviceHost;
    }
}

As you can see, this is the point at which I’m actually creating the container. When asked to create the service host, I simply create and return an instance of my custom service host and pass my container to it. My service host, then, takes care of wiring up the behavior and you already know the rest of that story.

public class WindsorServiceHost : WebServiceHost
{
    readonly WindsorContainer _container;

    public WindsorServiceHost(WindsorContainer container, Type serviceType, params Uri[] baseAddresses)
        : base(serviceType, baseAddresses) {
        _container = container;
    }

    protected override void OnOpening() {
        if (Description.Behaviors.Find<WindsorServiceBehavior>() == null)
            Description.Behaviors.Add(new WindsorServiceBehavior(_container));

        base.OnOpening();
    }
}

So, the chief complaint that I’ve heard about WCF is how painful the XML configuration is – here’s the great part – in WCF 4, the configuration is insanely straightforward – here’s the configuration to get this all up and going (keep in mind that I would actually like to externalize more of my current sample into configuration – but this should give you a pretty good sense of how much has been improved in this area).

<system.serviceModel>
  <serviceHostingEnvironment aspNetCompatibilityEnabled="true" />
  <standardEndpoints>
    <webHttpEndpoint>
      <standardEndpoint name="" helpEnabled="true" automaticFormatSelectionEnabled="true"/>
    </webHttpEndpoint>
  </standardEndpoints>
</system.serviceModel>

So deploying to IIS is as simple as building this into an *.svc file and writing a directive to point to my custom service host factory. But then I still end up with odd-looking URLs like http://localhost/SimpleWCFTestHost/PersonService.svc/Find/Howard. It would be even better if I could just drop that *.svc file and have completely clean URLs – just like I can do with MVC. Firstly, remember that MVC doesn’t actually give you clean URLs – it simply plugs into the new core routing engine that’s backed into ASP.NET 4. And that’s exactly the capability that WCF 4 gives you as well. To light this up, all I had to do was add this to my global.asax.cs file.

public class Global : HttpApplication
{
    protected void Application_Start(object sender, EventArgs e) {
        RegisterRoutes(RouteTable.Routes);
    }

    static void RegisterRoutes(RouteCollection routes) {
        routes.Add(new ServiceRoute("", new WindsorServiceHostFactory(), typeof(PersonService)));
    }
}

This simple entry and I now have the ability to address my service as http://localhost/SimpleWCFTestHost/Find/Howard. Had I wanted to give the service its own path segment, I could have specified my desired segment name as the first parameter to the ServiceRoute constructor.

After this experiment, WCF is at this point in time the more natural choice in my mind for building a general service layer. Note – I’m not saying that you shouldn’t use MVC to create services for the AJAX components on your pages. Just that if those services grow in scope such that they have other clients than your view, it might be time to look at a truer service platform – with WCF 4, it’s not *that* much more complicated.

Finally, I stumbled across a multi-part series that’s currently being developed on this topic over at The .NET Endpoint – It looks like they are just on part 4 of 12, so that’s only around the first 30 minutes of the movie – there’s still plenty of great stuff to come!

New Machine Parts List

hdierking — Sun, 10 Jan 2010 17:09:00 GMT

I guess there’s something in the water – I’ve now been bitten by the “upgrade my dev machine” bug. In my case, it really comes down to one very fundamental gap in my current setup – my > 3 yr old machine is still running a 32 bit proc which means that I can’t really take advantage of any improved virtualization technology. I had decided when I bought my current machine (Dell Dimension E510) that I was done building my own machines because of the potential time commitment and because I wasn’t convinced that I was really saving all that much money from buying parts vs. buying a machine. However, I also like to stretch the life of my machines as long as possible, and in looking around at Dell and Lenovo offerings, I’m not really seeing any offerings comparable to what I could buy/build – I’m thinking specifically around the processor and some little things like USB 3.0. Also, by building a machine, I can also cannibalize some of my current machine and upgrade those parts over time.

Anyways, here’s the list of parts I was looking at (I’ve also highlighted the parts I plan on reusing in the short term). I’m not a hardware guy so let me know if I’ve missed something or if there’s a better option.

parts	buy/reuse	price	qty	subtotal	buy link	Notes
case	buy	89.99	1	$89.99	http://www.newegg.com/Product/Product.aspx?Item=N82E16811133098	includes 450W power supply - 18.90" x 8.27" x 20.67"
case fan	buy			$0.00	N/A	N/A
cpu	buy	288.99	1	$288.99	http://www.newegg.com/Product/Product.aspx?Item=N82E16819115202	mboard req - LGA 1366 socket
cpu fan	buy	39.99	1	$39.99	http://www.newegg.com/Product/Product.aspx?Item=N82E16835207004	Sunbeam CR-CCTF 120 mm Core-Contact Freezer CPU Cooler W/TX-2
motherboard	buy	309.99	1	$309.99	http://www.newegg.com/Product/Product.aspx?Item=N82E16813131614	ASUS P6X58D Premium LGA 1366 Intel X58 SATA 6Gb/s USB 3.0 ATX Intel Motherboard
video card	reuse			$0.00		Had already upgraded
hard drive	reuse			$0.00
ram	buy	57.99	3	$173.97	http://www.newegg.com/Product/Product.aspx?Item=N82E16820104136	Kingston - 2GB
card reader	reuse			$0.00
dvd r/w	reuse			$0.00
Total				$902.93

More on Perceived Performance

hdierking — Mon, 14 Dec 2009 05:50:00 GMT

In my last post, I talked about some of the key takeaways (for me) after reading the book Designing and Engineering Time: The Psychology of Time Perception in Software. Unfortunately, I had to run out before I noted all of the items on my list, so I’m going to finish that list up here.

If performance is a differentiating factor between a function of your product and a similar function in a competitor’s product, you may not want to (or be able to) spend the time and money that it would take to make your product objectively faster. So how much faster do you need to worry about making your product in order to neutralize performance as a differentiating factor? According to the findings presented, researchers have determined that this bar is the geometric mean between the 2 performance measures. Users will associate a duration higher than this bar with the higher number and a duration lower than the bar with the lower number.
There are 2 important qualities to consider when collecting performance data – reliability and validity. Reliability, as it is explained, seems to be synonymous with repeatability (as a side note, when I look at our current measuring methods for MSDN and TechNet, reliability is one area where we need pretty dramatic improvement). Validity describes whether or not you’re actually measuring what you’re trying to measure. Since reliability and validity are independent of one another, a given method can fall into any of the 4 combinations of the 2.
Define your start and end events from the user’s point of view – not from whatever available options your tools give you. Not only does this give you a better picture of how your users perceive your application, but it means that you aren’t going to have to change your measuring method when you start optimizing the responsiveness characteristics of your application.
When measuring, choose an appropriate precision based on the responsiveness you’re shooting for. Just like performance tuning shouldn’t be some kind of open-ended “let’s just make everything as fast as possible” kind of exercise, it’s pointless to measure an Ajax call down to the milliseconds, since most of these operations will fall into the ‘continuous’ aka ‘user flow’ or even ‘captive’ interaction classes (for more on interaction classes, see the UI Timing Cheatsheet).
For assessing user tolerance, one technique is to have a user interact with several different prototypes of a given feature (in random order, even repeating some prototypes) and have them rate the experience along some scale of responsiveness. This can help to identify what are the key factors that influence tolerance.
One technique that I thought was pretty cool for managing a user’s perception of performance for a lengthy process (think product install) is to order the sub processes so that they proceed from longest duration to shortest duration. Research shows that users are more likely to watch the tail end of a process, so having the sub processes with shorter durations at the end leaves users feeling as though the overall process was faster (recall how annoying it is when the last 5% of an operation takes 95% of the time – see what I mean?)
Another technique that I thought was equally cool is one that the author calls “End on Time”. The basic premise is that rather than spend all of your energy trying to predict the end time with pinpoint accuracy, “Give approximate remaining time in time anchors while a process is running, but when the process is actually complete, add a few seconds and show the process counting down to and ending exactly at zero”. This is a great perception managing technique in that it creates the illusion of precision and builds confidence with your users. Keep in mind that the applicability here is for processes that are measured in terms of minutes – adding 3 seconds to a 2 second process will likely not create the desired perception.

So that wraps up my key takeaways from the book. Keep in mind that there is a ton of great stuff in there – I just pulled out some of the things that resonated with me. I’ll be trying to put some of this information to use over the next 6-9 months and will be sure to see how some of these techniques work in my own context. If you do the same, or if you’re currently using some similar techniques, do share.

Understanding Perceived Performance

hdierking — Sat, 12 Dec 2009 01:50:00 GMT

I recently finished reading the book Designing and Engineering Time: The Psychology of Time Perception in Software, by Steven Seow. If you care about performance (rhetorical – you should), you should definitely take a look at this book, as – as a developer, I know that I have tended to look at performance in a very 1-dimensional fashion. When I focused on performance, I did so without any well-reasoned goals (and as such without any well reasoned measures (measures were driven by whatever tools I had on hand)) – I just wanted to make everything faster. Conversely, when I wanted to be lazy about performance, I rationalized it in the way of building more expressive code (sometimes I got away with this rationalization – when I didn’t, it hurt).

Wouldn’t it have been better if I could have identified the human-computer interactions that my software enabled and then set real performance goals based on research in human cognitive psychology rather than on my own vague notions on “fast enough”? That’s what Seow’s book is all about – and in addition to the insights that he articulates, he also provides a plethora of references to the research that support those insights. What I’m going to here is just to outline a couple of the key points from the book – hopefully to entice you to read the book for yourself and in part to record my notes before I have to return the book to the library :)

Maister’s First Law of Service – satisfaction is the result of the difference between what the customer experiences (perceives) and what he/she expected (this is called disconfirmation)
The way that perception (and the things that influence perception) can predict things like adoption and use of technology can be illustrated in a model called the Technology Acceptance Model (TAM) – created by Fred Davis
There are three basic aspects of performance that influence a user’s overall experience, and as such there are three main approaches for managing performance-related issues:

Performance Dimension	Management Strategy
Actual Performance – the precise time durations for completing a task	Operational Management – purely technical strategy (e.g. change the way it works to make it faster)
Perceived Performance – the approximate duration for completing a task as experienced by the user	Perception Management – make the task feel like it takes less time (examples include distractions, progress indicators, etc.)
User Tolerance – based on expectations, the approximate duration threshold where anything beyond will be considered slow by a user	Tolerance Management – focus on making users more tolerant of a longer duration when it cannot be shortened or “disguised”

Vierordt’s Law describes how humans tend to overestimate short durations and underestimate long durations when asked to recall the duration of a recent activity
When thinking about performance from the user’s perspective, it’s more productive to think about responsiveness in terms of various buckets based on the user’s expectations. The book introduces interaction classes to help categorize these various types of responsiveness. Actually, the author provides these details for free in his UI Timing Cheatsheet. I’m not going to talk about interaction classes more here because Seow does a much better job in this doc than I could do here anyway.
The 20% rule is an average Weber ratio which basically says that users notice a difference in performance of a function/feature when difference is more than 20% of the previous performance.
Users don’t estimate time in terms of actual numbers (e.g. 4.67 seconds). Instead, they tend to gravitate towards “Time Anchors” (e.g. 1, 2, 3, 5, 10, 15, etc.) – see the cheatsheet for more details about the time anchor matrix.

I’ve gotta run and catch my bus, but when I get some time, I’ll follow this up with some of my book notes on measuring time. However, like I said, I would definitely recommend you checking out this quick read.

Update: the follow up can be found here

Astoria, SSIS Adapters and BDD – Projecting Projections

hdierking — Wed, 18 Nov 2009 22:19:00 GMT

The big idea behind ADO.NET Data Services is that it enables a data source (generally a relational data source) to be exposed via a RESTful service. In the context of fetching data – which is the principle concern of our SSIS data source component, data is returned as either an ATOM feed or as a JSON list. Despite the particular data format, the general principles are the same – you’re taking a relational model and projecting it in a more serial structure. Now, if all you’re querying is a single table, it doesn’t matter all that much – you’re simply taking a tabular data set and serializing each row as an entry in the ATOM feed. However, ADO.NET Data Services provides an expansion feature that traverses a relationship and inserts the child data inline in the resulting feed. Keep in mind that the ADO.NET Data Services API is based on HTTP – and the expansion feature is exposed simply as a querystring parameter.

Looking At the Data

Consider the following query: http://localhost:7558/northwind.svc/Products

This will return the following ATOM feed (fragment):

<feed xml:base="http://localhost:7558/Northwind.svc/" 
   xmlns:d="http://schemas.microsoft.com/ado/2007/08/dataservices" 
   xmlns:m="http://schemas.microsoft.com/ado/2007/08/dataservices/metadata" 
   xmlns="http://www.w3.org/2005/Atom">
   <title type="text">Products</title> 
   <id>http://localhost:7558/northwind.svc/Products</id> 
   <updated>2009-11-17T22:45:22Z</updated> 
   <link rel="self" title="Products" href="Products" /> 
   <entry>
      <id>http://localhost:7558/Northwind.svc/Products(1)</id> 
      <title type="text" /> 
      <updated>2009-11-17T22:45:22Z</updated> 
      <author>
         <name /> 
      </author>
      <link rel="edit" title="Products" href="Products(1)" /> 
      <link rel="http://schemas.microsoft.com/ado/2007/08/dataservices/related/Categories" 
         type="application/atom+xml;type=entry" 
         title="Categories" 
         href="Products(1)/Categories" /> 
      <link rel="http://schemas.microsoft.com/ado/2007/08/dataservices/related/Order_Details" 
         type="application/atom+xml;type=feed" 
         title="Order_Details" 
         href="Products(1)/Order_Details" /> 
      <link rel="http://schemas.microsoft.com/ado/2007/08/dataservices/related/Suppliers" 
         type="application/atom+xml;type=entry" 
         title="Suppliers" 
         href="Products(1)/Suppliers" /> 
      <category term="NorthwindModel.Products" 
         scheme="http://schemas.microsoft.com/ado/2007/08/dataservices/scheme" /> 
      <content type="application/xml">
         <m:properties>
            <d:Discontinued m:type="Edm.Boolean">false</d:Discontinued> 
            <d:ProductID m:type="Edm.Int32">1</d:ProductID> 
            <d:ProductName>Chai</d:ProductName> 
            <d:QuantityPerUnit>10 boxes x 20 bags</d:QuantityPerUnit> 
            <d:ReorderLevel m:type="Edm.Int16">10</d:ReorderLevel> 
            <d:UnitPrice m:type="Edm.Decimal">18.0000</d:UnitPrice> 
            <d:UnitsInStock m:type="Edm.Int16">39</d:UnitsInStock> 
            <d:UnitsOnOrder m:type="Edm.Int16">0</d:UnitsOnOrder> 
         </m:properties>
      </content>
   </entry>
   ...
</feed>

As you can see, for a single resource query, the resulting feed is pretty simple with a feed consisting of multiple entries. Each entry (in addition to the basic metadata) contains properties for the actual row values and links for relationships. To leverage the expansion functionality – in this case, we’ll expand Suppliers, we would modify the query to look like the following: http://localhost:7558/northwind.svc/Products?$expand=Suppliers

From this query, we would get the following, expanded result:

<entry>
   <id>http://localhost:7558/Northwind.svc/Products(1)</id> 
   <title type="text" /> 
   <updated>2009-11-17T23:04:46Z</updated> 
   <author>
      <name /> 
   </author>
   <link rel="edit" title="Products" href="Products(1)" /> 
   <link rel="http://schemas.microsoft.com/ado/2007/08/dataservices/related/Categories" 
      type="application/atom+xml;type=entry" title="Categories" href="Products(1)/Categories" /> 
   <link rel="http://schemas.microsoft.com/ado/2007/08/dataservices/related/Order_Details" 
      type="application/atom+xml;type=feed" title="Order_Details" href="Products(1)/Order_Details" /> 
   <link rel="http://schemas.microsoft.com/ado/2007/08/dataservices/related/Suppliers" 
      type="application/atom+xml;type=entry" title="Suppliers" href="Products(1)/Suppliers">
      <m:inline>
         <entry m:etag="W/"'49%20Gilbert%20St.','London','EC1%204SD'"">
            <id>http://localhost:7558/Northwind.svc/Suppliers(1)</id> 
            <title type="text" /> 
            <updated>2009-11-17T23:04:46Z</updated> 
            <author>
               <name /> 
            </author>
            <link rel="edit" title="Suppliers" href="Suppliers(1)" /> 
            <link rel="http://schemas.microsoft.com/ado/2007/08/dataservices/related/Products" 
               type="application/atom+xml;type=feed" title="Products" href="Suppliers(1)/Products" /> 
            <category term="NorthwindModel.Suppliers" 
               scheme="http://schemas.microsoft.com/ado/2007/08/dataservices/scheme" /> 
            <content type="application/xml">
               <m:properties>
                  <d:Address>49 Gilbert St.</d:Address> 
                  <d:City>London</d:City> 
                  <d:CompanyName>Exotic Liquids</d:CompanyName> 
                  <d:ContactName>Charlotte Cooper</d:ContactName> 
                  <d:ContactTitle>Purchasing Manager</d:ContactTitle> 
                  <d:Country>UK</d:Country> 
                  <d:Fax m:null="true" /> 
                  <d:HomePage m:null="true" /> 
                  <d:Phone>(171) 555-2222</d:Phone> 
                  <d:PostalCode>EC1 4SD</d:PostalCode> 
                  <d:Region m:null="true" /> 
                  <d:SupplierID m:type="Edm.Int32">1</d:SupplierID> 
               </m:properties>
            </content>
         </entry>
      </m:inline>
   </link>
   <category term="NorthwindModel.Products" 
      scheme="http://schemas.microsoft.com/ado/2007/08/dataservices/scheme" /> 
   <content type="application/xml">
      <m:properties>
         <d:Discontinued m:type="Edm.Boolean">false</d:Discontinued> 
         <d:ProductID m:type="Edm.Int32">1</d:ProductID> 
         <d:ProductName>Chai</d:ProductName> 
         <d:QuantityPerUnit>10 boxes x 20 bags</d:QuantityPerUnit> 
         <d:ReorderLevel m:type="Edm.Int16">10</d:ReorderLevel> 
         <d:UnitPrice m:type="Edm.Decimal">18.0000</d:UnitPrice> 
         <d:UnitsInStock m:type="Edm.Int16">39</d:UnitsInStock> 
         <d:UnitsOnOrder m:type="Edm.Int16">0</d:UnitsOnOrder> 
      </m:properties>
   </content>
</entry>

While this example demonstrates expansion of a single link resource, ADO.NET Data Services allows you to expand a single link, multiple links, or multiple nested links. The funny thing about this is that the SSIS data flow engine takes much more of a relational view of the world with its stream architecture. This means that we’re starting with a relational model, projecting into a hierarchical model, and then projecting back to a relational model. The question, then, is how we go about designing (and testing) this projection.

To start, here’s a few high level design principles:

Every resource usage (primary and all expanded) should yield a unique output stream in an SSIS data flow which is visible in the data flow designer
Composition of output streams (e.g. joining) should be done via data flow designer transform components

A Unique Stream Per Resource

Every resource that is returned as part of an ADO.NET Data Services query, whether it is the primary resource (e.g. Products) or an expanded resource, should have a unique output in the data flow designer. Moreover, if a resource is referenced in more than 1 role (e.g. http://localhost:7558/northwind.svc/Products?$expand=Suppliers/Products), an output stream should be created for each role. The goal of having multiple output streams is that each one can then by manipulated independently using the data flow designer.

Enabling Composition

One of the primary reasons behind creating unique streams per resource *role* is to enable granular composition/branching inside the data flow designer (as shown above). The service metadata provided by a data service provides a type definition for each resource managed by the service. This makes it initially straightforward for determining data shape from a data services query. However, in order to effectively project the parent/child relationship specified with the expansion feature, we need to make sure we include the identifiers of the parent record in context of the child record. I started with the following specifications:

public class when_employees_resource_query_expands_territories : with_northwind_CSDL
{
    Establish context = () => {
        queryBuilder.ResourcePath = "Employees";
        queryBuilder.OperationsString = "$expand=Territories";
    };

    Because of = () => query = queryBuilder.GetQuery();

    It should_declare_2_resources = () => query.DataShape.Count().ShouldEqual(2);

    It should_have_1_reference_property_for_territories =
        () => query.DataShape.ElementAt(1).Properties
            .Where(p => p.GetType() == typeof (RefProperty)).Count()
            .ShouldEqual(1);

    It should_have_2_properties_for_territories =
        () => query.DataShape.ElementAt(1).Properties
            .Where(p => p.GetType() == typeof (Property)).Count()
            .ShouldEqual(2);

    It should_have_correct_query_string = 
        () => query.ToString().ShouldEqual(
            string.Format("{0}/Employees?$expand=Territories", Constants.SERVICE_URI));

    It should_have_employees_as_first_resource = 
        () => query.DataShape.ElementAt(0).Name.ShouldEqual("Employees");

    It should_have_territories_as_the_second_resource =
        () => query.DataShape.ElementAt(1).Name.ShouldEqual("Territories");
}

First, note the with_northwind_CSDL base class. Aaron brought up in an earlier post that this “with…” convention was one that he was actually moving further away from, so in this point, I want to call out more specifically what is happening here. The basic point of this shared context is to setup a mock reader to get the XML metadata from a file rather than from the $metadata clause, which is what is used at runtime.

public class with_northwind_CSDL
{
    const string NORTHWIND_METADATA_PATH = "svc_metadata.xml";
    protected static DataServicesQuery query;
    protected static IDataServicesQueryBuilder queryBuilder;

    Establish context = () => {
        var mockMetadataReader = new Mock<IReader>();
        mockMetadataReader.Setup(r => r.GetData(Constants.SERVICE_URI + "/$metadata"))
            .Returns(new StreamReader(NORTHWIND_METADATA_PATH));

        queryBuilder = new DataServicesQueryBuilder(new CSDLResourceBuilder(), 
            mockMetadataReader.Object) {ServiceURI = Constants.SERVICE_URI};
    };
}

Setting this up in a base class rather than in every spec class saved me a fair amount of repetitive typing – especially since my specifications here are pivoting on how the query shapes the data shape rather than on how the metadata is acquired.

So once I had my basic specs setup, all that remained was the implementation. As you can see from the following, the basic task of creating the data shape is effectively divided among 3 types:

the data reader is responsible for getting the metadata from some source (in the running service, this is done using a HTTP GET request – for my specs, this was mocked with an XML file reader)
the resource builder is responsible for producing resources (the main elements of the data shape) from the metadata provided by the data reader
the query builder is responsible for knowing how to interpret the query URI and ensuring that parent/child projections are correctly constructed.

Based on this separation of roles, you can see that the bulk of the projection logic resides in the query builder class – specifically in the GetResources method. This logic looks like the following:

IEnumerable<Resource> GetResources() {
    var metadata = _metadataReader.GetData(string.Format("{0}/$metadata", ServiceURI));
    _resourceBuilder.Initialize(metadata);

    // initialize resourcs collection with primary resource
    var resources = new List<Resource> {
                                           _resourceBuilder.GetResource(ResourcePath.Split(new[] {'/'}).Last())
                                       };

    //add resources specified by the presence of an $expand query operation
    if (!string.IsNullOrEmpty(OperationsString)) {
        var expandOperation = QueryOperation.ParseQueryOperations(OperationsString).Expand();

        foreach (var expandedPath in expandOperation.ResourcePaths)
            if (expandedPath.Contains("/")) {
                var pathAxes = expandedPath.Split(new[] {'/'});
                for (var i = 0; i < pathAxes.Length; i++)
                    resources.Add(_resourceBuilder.GetResource(pathAxes[ i ], resources.Last()));
            }
            else
                resources.Add(_resourceBuilder.GetResource(expandedPath, resources.Last()));
    }

    return resources;
}

As you can see, this method is basically getting resource names from the data services query string and then asking the resource builder to build those resources. For expanded resources, the builder is asked to produce the resource incorporating the keys of the parent resource.

In sum, I think that dealing with $expand is one of the more interesting aspect of this component, so I wanted to spend a bit more time going through it. To see all of the components working together, you can download the entire source code at http://code.msdn.microsoft.com/ssisastoria. But in sum, the problem looks like: DB (relational) –> Data Services (hierarchical) –> SSIS (relational).

Astoria, SSIS Adapters and BDD – Class Design

hdierking — Wed, 18 Nov 2009 20:11:00 GMT

In the process of writing a post about one specific feature of the design, I realized that I hadn’t actually described the moving parts in the solution. There are 3 projects in the VS solution: the SSIS-specific code, the core application logic, and the application logic tests (specifications). I’ll talk more about the SSIS part later and focus here on the core application logic and tests.

At a high level, there are 2 major logical components that interact to support SSIS at design time and runtime – those are DataServicesQuery and DataServicesQueryBuilder.

The DataServicesQueryBuilder Object

The query builder is responsible for getting the metadata from ADO.NET Data Services and constructing the resources that are used by SSIS during the design time experience (those resources are also passed to the query to be used at runtime for making sure that a data element gets put in the right place in the buffer). The basic elements of the query builder are shown below:

As you can see, responsibilities are further decomposed as follows:

The DataServicesQueryBuilder class is responsible for understanding an ADO.NET Data Services query and producing a query that can be consumed by the SSIS runtime
The IReader interface is responsible for getting stream data from some data source – the concrete implementation used by the query builder is the HttpRequestReader. This implementation is used by the query builder to get service metadata as follows:

var metadata = _metadataReader.GetData(
    string.Format("{0}/$metadata", ServiceURI));

The IResourceBuilder interface is responsible for producing Resource instances from whatever stream data is handed to it. Since ADO.NET Data Services currently has a dependency on EDM (more specifically, CSDL), the only relevant concrete class at the moment is CSDLResourceBuilder.

The only other thing I want to point out from the query builder classes is that we have a class for both Property and for RefProperty. Resource properties are the equivalent of columns (if looking at the data from a tabular perspective) and RefProperties are the equivalent of links. This will become more obvious when I discuss use of the $expand keyword in a future post, but the difference between parent/child (and subsequently, the driver behind inheritance) is based on the following runtime behavior (when extracting data from results data):

public class Property
{
    ...

    public virtual object GetValue(XElement element) {
        var propertyElement = element.Element(XName.Get("content", XmlNamespaces.AtomXmlNamespace))
            .Element(XName.Get("properties", XmlNamespaces.DataServicesMetadataXmlNamespace))
            .Element(XName.Get(Name, XmlNamespaces.DataServicesXmlNamespace));

        var propertyVal = propertyElement.Value;

        var val = Convert.ChangeType(propertyVal, PropertyType);

        return val;
    }
}

public class RefProperty : Property
{
    public RefProperty(string name, Type type, int? maxLength, bool? unicode)
        : base(name, type, maxLength, unicode) { }

    public override object GetValue(XElement element) {
        return element.Parent.Name.LocalName=="feed" 
            ? base.GetValue(element.Parent.Parent.Parent.Parent) 
            : base.GetValue(element.Parent.Parent.Parent);
    }
}

The DataServicesQuery object

Ultimately, the SSIS code simply uses the query builder in order to get a DataServicesQuery object. The query builder configures the query with the collection of resources that it created (called the data shape). The query is responsible for executing the actual query and extracting the results from the returned XML (ATOM feed) into a tabular format which can be used to populate SSIS output buffers. As you can see below, the query class layout is much simpler than the builder:

As you can see, the reader is again used to pull down stream data. The query then uses it’s data shape (resources) to take the XML data stream and create a ResultSet for each resource. 1..n ResultSets are then handed back as the result of executing the query – this is what gets handed back to SSIS and allows the SSIS adapter code to populate the SSIS buffers.

You can download and look at the complete source at http://code.msdn.microsoft.com/ssisastoria. As always, please let me know if there’s stuff that doesn’t make sense or that could be refactored to be improved.

Astoria, SSIS Adapters and BDD – On BDD and MSpec

hdierking — Fri, 13 Nov 2009 01:04:00 GMT

Let me preface this post by saying that it is about my (early) experiences practicing BDD in the context of a specific specification framework – MSpec. There are plenty of folks out there who are much more authoritative on this subject than I. And if you happen to be one of those people, I would love to get your feedback on what I’ve done here and how I could do it better.

In developing a SSIS data source to talk to an ADO.NET Data Services endpoint, shifting my approach from a basic TDD style to specifications was not originally part of the plan. However, as I got deeper into development, I felt as though I actually wanted a bit more rigor in terms of semantic restrictions (e.g. arrange/act/assert) than I was getting out of the box from my unit test framework. MSpec had been on my radar for a little while (I was introduced to both this framework and BDD in general at ALT.NET), so I took a quick look - and after staring oddly for a moment at the =()=> syntax decided that it provided the kind of semantic consistency I was really looking for. Additionally, the fact that it could plug into the ReSharper unit test runner (with underscores removed from boxcar casing no less!) made it quick and very easy to get up and going on the journey writing specifications rather than tests. Going through the specification writing exercise on this project has me convinced at this point that as someone else once said, BDD is TDD done right. I certainly plan to keep pursuing specifications over traditional unit tests in future efforts.

So where to start?

I’m not going to spend time in this post talking about how to use MSpec – there’s already lots of good guidance out there. That said, because the syntax initially looks a bit funky, I will say 2 quick things that are helpful to keep in mind when you first look at specifications built on top of MSpec.

Everything in your specification (test) class is a field. And all of the work gets down via delegate fields which are of types with names like “Establish”, “Because”, and “It”
Regarding the =()=> syntax, you are probably already comfortable with lambda expressions, so you can ignore the => on the right - and you can get the assignment operator on the left. Therefore, that just leaves the empty parenthetical – and this just means that your delegate (lambda) is for a function that has no arguments. There, all better?

I already had some unit tests written – though some of them were more of integration tests and revealed some pretty serious problems in my design – but we’ll deal more with those topics later. So my starting point was to try and re-write some of my existing tests as specifications. For example, take the following:

[Test]
public void Given_Null_URI_Throw_Exception() {
    try {
        new DataServicesQuery(_dataServicesQueryBuilder, null, "Suppliers");
    }
    catch(ArgumentNullException ex) {
        Assert.AreEqual("serviceUri", ex.ParamName);
    }
}

Re-writing this as a specification resulted in the following:

public class when_service_uri_is_null : with_dummy_resource_builder_and_reader
{
    static NullReferenceException lastException;

    Because service_uri_is_null = () => {
        try {
            queryBuilder.ServiceURI = null;
            queryBuilder.ResourcePath = "Products";
            queryBuilder.GetQuery();
        }
        catch (NullReferenceException ex) {
            lastException = ex;
        }
    };

    It should_have_specific_error_message = 
        () => lastException.Message.ShouldEqual("Service URI must be provided");

    It should_raise_an_exception = 
        () => lastException.ShouldNotBeNull();
}

A couple things to point out here.

First and foremost, there is a more rigid semantic that is enforced by both convention and even by the type system. The convention is that classes follow a “when X [with Y]” pattern (I found that this class name should map very tightly to the instructions contained in the Establish and/or Because expressions – if it does not, you may have a problem in your design). Also, assertion (“It” expressions) delegates should follow a naming convention whereby “should …” is followed by a description of the assertion. MSpec also uses the type system to enforces this convention in how the delegate types are named (e.g. – “Because”, “It”).
All of the NUnit attributes identifying test fixtures and tests are gone. MSpec is smart enough to figure out what is a test vs. what is a regular old class based on the MSpec delegates that make up the class. When looked at through the NUnit lens, the specification class remains the test fixture, and each “It” delegate instance becomes an NUnit test.
Notice the with_dummy_resource_builder_and_reader class. As with unit tests, inheritance can be a great way to provide shared context to multiple test fixtures. This base class is responsible for 2 things: 1) exposing a few data fields for its child classes, and 2) providing its own Establish delegate to setup those data fields.
For me, this format of specifications seemed to yield another benefit – without explicit effort. That benefit was more tests with each test having a much better definition around its responsibility.

This particular type of test – one where you’re expecting an exception – was one where I wasn’t 100% confident in my approach. However, I didn’t find any baked in support from MSpec – and frankly, for the framework to provide a feature similar to NUnit’s ExpectedExceptionAttribute doesn’t really make a lot of sense to me for writing specifications anyway. You can see above the approach that I took – simply trap the exception and then run a couple tests over the exception to assert various characteristics of it. If there’s a better way to go about this, I would love to hear it.

After I started getting the hang of writing specifications by rewriting my tests, I began venturing into new parts of my application – and this is where I really started realizing just how much ceremony exists in traditional unit testing frameworks compared to something like MSpec. Because mostly everything in MSpec is a field, there’s no need to provide any kind of method body, etc. after declaring the title of the specification. So, for example, when I wanted to write specifications around executing a query, I was able to quickly capture the specs as follows:

public class when_query_is_executed_for_products_with_expanded_suppliers
     : with_query_shape_factory
{
    It should_have_2_result_sets;
    It should_have_a_result_set_named_products;
    It should_have_8_columns_in_products_result_set;
    It should_have_77_rows_in_products_result_set;
    It should_have_a_result_set_named_suppliers;
    It should_have_13_columns_in_suppliers_result_set;
    It should_have_77_rows_in_suppliers_result_set;
}

Remember, “It” is simply a type and “should_have_2_result_sets” is simply a field declaration – no need to initialize it with anything just yet. When the MSpec runner is executing, it checks to see whether the field has been initialized – if it hasn’t, as in this case, it flags it as if you used NUnit’s IgnoreAttribute.

Finally, I mentioned earlier in this post about using ReSharper to run MSpec specifications. While I would probably still be interested in MSpec if this integration didn’t exist, the fact that it was there removed any question or hesitation. We talk a lot about the readability of tests, and of tests being the spec – and this is, based on what I’ve experienced so far, where specification frameworks like MSpec really shine. Bringing that goodness directly into Visual Studio ala ReSharper mades the feedback loop even tighter, and really made for a happy development process.

If you’ve read this far and you have no clue what this project is that I’m talking about, see the overview here. If you want to download the code and look at it yourself, check it out at http://code.msdn.microsoft.com/ssisastoria

Using Latest NUnit Version With MSpec

hdierking — Sun, 08 Nov 2009 20:11:00 GMT

As I’ve been working another blog post about my adventures in learning BDD ala MSpec, I ran into something that I wanted to share as the problem wasn’t immediately obvious (to me anyway).

The problem is that MSpec has a dependency on version 2.4.6.0 of the nunit.framework assembly – and frankly, it’s not really a very big problem. In fact, I didn’t even notice it until now because I had been running all of my specs exclusively through the Resharper runner (enabled by the MSpec plugin).

** As a side note, using R# with the MSpec plugin absolutely rocks, and while the runner is needed for integrating MSpec with your automated builds, I would totally recommend R# for your regular dev rhythm.

In my project, I originally had some integration tests in the same assembly as my specifications. As a result, I had a direct reference to NUnit (version 2.5.2.9222) in addition to the reference to Machine.Specifications and Machine.Specifications.NUnit (which has a reference to version 2.4.6.0). As a result, the later version of NUnit was being copied to my bin folder and throwing a file load exception for every specification when run using the console runner.

My initial reaction was to simply move all my integration tests to a different assembly and remove the direct NUnit reference in my specifications assembly (remember that referencing Machine.Specifications.NUnit will copy the version of NUnit to the bin folder that MSpec expects. No problem – everything works as you would expect.

However, it seems reasonable to me that, at least in larger projects that are wanting to incorporate specs and not have to re-write all existing tests, that it would be good to simply reconfigure the MSpec console runner to use the latest version of NUnit. This can be done easily using the standard .NET assembly binding redirection feature. Simply create a file named Machine.Specifications.ConsoleRunner.exe.config and drop it into the MSpec folder that contains your console runner (Machine.Specifications.ConsoleRunner.exe). The contents of the file should look like this.

<?xml version="1.0"?>
<configuration>
    <runtime>
      <assemblyBinding xmlns="urn:schemas-microsoft-com:asm.v1">
        <dependentAssembly>
          <assemblyIdentity name="nunit.framework"
                            publicKeyToken="96d09a1eb7f44a77"
                            culture="neutral" />
          <bindingRedirect oldVersion="2.4.6.0"
                           newVersion="2.5.2.9222"/>
        </dependentAssembly>
      </assemblyBinding>
    </runtime>
</configuration>

Run the MSpec console again, and voila! Works like a charm.

Astoria, SSIS Adapters and BDD – Code

hdierking — Fri, 06 Nov 2009 19:57:00 GMT

I’m currently working on the post talking about how, in the course of writing my SSIS data source for ADO.NET Data Services, I stumbled into replacing my unit tests with specifications. However, in advance of that, I wanted to go ahead and provide the link to the source code. I would love to get feedback on what’s working and what’s not so that I can get better – and so that ideally, those the random stranger who downloads and looks at it on some idle Tuesday gets better as well.

The code can be found at http://code.msdn.microsoft.com/ssisastoria

Getting Your Groove Off

hdierking — Thu, 29 Oct 2009 21:26:00 GMT

Until recently, the only knowledge I had of Groove was that it was some kind of collaboration product that we got as a result of bringing in Ray Ozzie. Then I got a new manager who insisted that our team standardize on Groove for the team document repository - so I got setup on Groove and went on with my life. In general, there’s nothing I found terribly impressive about it, but it wasn’t really getting in my way so no reason to get too upset…

Until my Tortoise SVN icon overlays stopped working.

I ended up spending quite a bit of time this morning trying to figure out exactly why they weren’t working. It was odd – I had no problems on my home machine – but on both of my work machines, same result – no TSVN icons. Finally, @pilotbob pointed me to this article which set off a chain of recollections – most notably that the major difference between my work machines and my home machine had to do with the order that I installed stuff. On my home machine, I installed Office after I got my dev environment setup. However, on my work machine, Office was installed as a part of the default OS install image - and Groove was included in that installation.

So why is this important? Because if you remember from the article, Windows supports only 15 overlay handler slots – and installing Groove puts 5 handlers right there on top of my TSVN handlers – the net result being that my TSVN icon overlays don’t load.

To fix, I simply went into the registry, deleted all of the Groove overlay handlers from HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\CurrentVersion\Explorer\ShellIconOverlayIdentifiers

Then rebooted. Voila! TSVN icon overlays come right back up!

Astoria, SSIS Adapters and BDD – Take 1

hdierking — Wed, 28 Oct 2009 16:00:17 GMT

Yea I know – 3 things you never thought you would hear mentioned together. And to be really honest, I didn’t start out trying to build this in a BDD fashion – and since I’m totally new to the disciplines of BDD, I may not have been all that successful. But hey, it’s all about the journey, and so I wanted to at least go ahead and start posting about the work I’ve been doing here, partly to share and partly to get your input ways to improve my design/specs.

So here was the initial goal – I was studying ADO.NET Data Services (previously known as Astoria) when I remembered this article that I once read in the Architecture Journal about using RSS as a technique for surfacing data to load into an enterprise data warehouse. That led me to think that it would be a cool opportunity to simultaneously learn Astoria and feed my BI sweet tooth by writing a SSIS data source that could consume an Astoria endpoint. Given the fact that more and more systems are exposing data using syndication, it seemed like creating an SSIS data source was something that could be helpful beyond my simply wanting to play with the technology.

Wait - ADO.NET Data Services what??

Just in case you weren’t 100% sure what ADO.NET Data Services is, it’s technology to expose data and enable operations on data via RESTful operations. You can find out much more about it here and here, but for the purposes of this exercise, here are the 3 things you need to know.

It uses all standard HTTP operations – which means that for getting data, you simply format a HTTP GET request for the service URL plus additional path information and optional querystring arguments to tell the framework how to shape the data.
While there’s not a dependency on the Entity Framework, there is a dependency on the Entity Data Model in that metadata is exposed as CSDL.
Though you have a couple formatting options when getting data, the primary one (and the one used by the data source component) is to get the data as an ATOM feed. The data source component can then project the XML into the appropriate output buffer (or buffers as you’ll see later).

Wait – SSIS what??

Sql Server Integration Services is Microsoft’s ETL tool that ships with SQL Server. You can start learning more about it here, but for the purpose of this exercise, here are the 3 things you need to know.

The unit of execution in SSIS is the package. This is really similar nomenclature to DTS because, not surprisingly, SSIS evolved out of DTS.
Every package contains 2 kinds of logic: 1 “Control Flow” and zero-to-many “Data Flows”. Control flow is meant for larger operations like copying files or running some kind of database maintenance operation (and while interesting, it’s not something I care about for this exercise). The real meat of SSIS (IMHO) happens within a data flow – this is where data is extracted from a source, massaged, and put into a destination. Data flows maintain the entire pipeline of operations between source and destination via in memory buffers.
Data flows use data source components to extract data from the source and add it to the appropriate buffers in the pipeline. Data source components are also used to provide metadata to the SSIS design-time experience so that package designers can use it to clean up and massage the data using transformation components (** for those experienced SSIS folks, it’s worth noting that there is an XML data source component that is a part of the product, and since ADO.NET Data Services returns data as an ATOM feed, is is absolutely possible to consume it directly. However, in order to take this approach, you will need to generate a XSD schema and associate it with the component at design time for the metadata – and every time you change the shape of your data services query, you will have to re-generate and re-associate a new schema. This was one of the primary motivators for me to create a custom data source).

So then, the initial goals were as follows -

Create an SSIS data source component that could consume an ADO.NET Data Services endpoint
Ensure that the component participated in both the design time and runtime experiences
Ensure that the component was able to handle projections of ADO.NET Data services queries that took advantage of the $expand clause

As I got into the development I was pretty quickly able to accomplish the first 2 goals. For example, given the following query configuration..

I was able to get the data shape using ADO.NET Data Service’s $metadata option and expose it back to the designer..

I was even able to run the package for a simple data services query and pipe the results to a flat file..

However, when I started implementing that 3rd goal of managing projections, I quickly realized my quick and dirty project need more thought in its design. This drove me to add a few more goals.

Project should separate the core logic of the component from the SSIS infrastructure – which is heavily reliant on inheritance
Core component logic should be independently testable without requiring a running data services endpoint

Separating the logic out into a different assembly wasn’t really all that painful – just a few keystrokes of Resharper, a couple file moves, a reference added, and I was good to go. However, because this started as a quick and dirty project (I should have known better), I hadn’t designed for data source independence and as such had some larger refactorings to do. And in that process (** keep in mind that this started as an remains primarily a learning exercise for me – I would NEVER recommend taking this kind of “while I’m at it approach” on a real project), I decided that my tests weren’t really where I wanted them and that I had always wanted to get my hands dirty with BDD anyway, so why not? It’s just one more goal.

Tests should be written as specifications

I’ve read from some that BDD is really meant more for testing larger behaviors in business applications. However, it feels pretty good (though initially uncomfortable) at the grain at which I’m using it. For example..

public class when_products_resource_is_queried : with_northwind_CSDL
{
    Establish context = () => queryBuilder.ResourcePath = "Products";

    Because of = () => query = queryBuilder.GetQuery();

    It should_have_8_properties = () => 
        query.DataShape.First().Properties.Count.ShouldEqual(8);

    It should_have_a_single_resource = () => 
        query.DataShape.Count().ShouldEqual(1);

    It should_have_1_key_property = () => 
        query.DataShape.First().Keys.Count.ShouldEqual(1);

    It should_have_correct_query_string = () =>
        query.ToString().ShouldEqual(string.Format(
            "{0}/Products", Constants.SERVICE_URI));
}

Ok – so those were the goals as they evolved. Going forward, my plan is to dig into the logic design and specifications, then talk specifically about projections, then finally talk about how this stuff works with SSIS (and how I would like to see the SSIS API simplified). Sorry to not get into all that tonight, but as my friend Glenn has advised me countless times, if you wait to blog until everything is nice and tidy and completed, you may never blog again.

Goodbye MSDN Magazine

hdierking — Wed, 23 Sep 2009 05:28:00 GMT

I am a huge fan of MSDN Magazine (and it’s predecessor publications going back to MSJ). There are just a handful of resources that have been so constant throughout the bulk of my career and MSDN Magazine is unquestionably one of those. This is why when Matt Carter pinged me over 2 years ago to tell me that there was an opening for the Editor-in-Chief role, I jumped at the opportunity without a moment of hesitation. And it’s been a great and wild ride these last couple of years. More importantly, I think that we’ve done some good in trying to improve the way that people think about building and constructing software. It’s always a tricky balancing act trying to cover new technologies and simultaneously trying to help people do a better job with the technologies they are already using, but I’m pleased with some of the directional shifts we made as a magazine favoring practices over technology – and really, those shifts wouldn’t have been possible without so many people like Jeremy Miller, Dave Laribee, James Kovacs, Ayende Rahien (December spoiler!), and so many other from what began as ALT.NET stepping up and writing for us. I hope that this is indicative of a trend that continues long after my departure.

So then why leave? In the end, I just came to the realization that the passion that initially drove me to the magazine was for creating software – not running a magazine business. And as the business elements became more and more consuming – and as the time that they occupied pushed out my ability to stay close to the technology, I came to conclude 2 things – 1) I was finding it harder to sustain the passion I came to the role with (because the object of that passion was fading) and 2) I was losing skill at an alarming rate. Summary – I needed to move back into the engineering world. As such, when a role opened up on the MSDN and TechNet Web platform team, I decided to move. There’s a lot of work to be done on those websites (a LOT) and most of the current work is around the more aesthetic elements of the UX. These things are certainly good and necessary, but I still don’t think they are anything remotely close to what’s necessary to make the sites truly helpful. In my new role, I’ll be focusing on driving “intelligent Web” innovations into the platform – so at the very least, I’ll get to play with some really cool tech..

So to wrap it up, it’s been awesome – but now time to try something different – and maybe finally have time to blog again!

Windows 7 Boot from VHD – A Few Gotchas

hdierking — Sat, 08 Aug 2009 17:33:21 GMT

When I first read about the capability in Windows 7 to natively boot a VHD, I was ecstatic – finally I could install all of my beta software in a sandboxed environment without having to put up with the performance cost of virtualized hardware. So I read through several posts such as this one from Scott and then proceeded to modify my boot record accordingly – pointing it at a vhd that I had already created via Windows Virtual PC. As those of you who have played with this already know, it’s not quite that simple, and it took me a little bit of time and digging to figure out exactly where the problems were – so I want to take a second and list out the major gotchas that I ran into – hopefully this will save you some extra digging.

By default, Windows Virtual PC puts both your vhd and vm config files in “C:\Users\[you]\AppData\Local\Microsoft\Windows Virtual PC\Virtual Machines” – this folder has restricted permissions and while pointing your master boot record at a vhd file found in this location will not raise an error when running bcdedit, it will rear its ugly head when you reboot. And while the error is recoverable and harmless, it looks a bit scary – something about the master boot record database being corrupted – yikes!
You must run sysprep on the vm that you intend to boot from – and you need to use the “generalize” option. Otherwise, your system will blue-screen when trying to boot your vhd because it will try and use the virtual machine drivers and not auto-detect the drivers for the physical hardware.
(this was the one that gave me fits) Pay attention to the size of your vhd file, even if you’re creating a dynamically sized disk. By default, the disk size for a dynamically expanding disk (when created through Windows Virtual PC) is 130 GB. If this size exceeds the amount of available storage on your physical disk, your vhd will start to boot and then blue-screen. Fortunately, the stop error is actually comprehensible – it will tell you that you don’t have enough physical storage to enable the vhd to expand to its maximum capacity (or something equivalent). So in short, be explicit in configuring your vhd, and make sure that the vhd size doesn’t exceed available physical storage limits (even on dynamically expanding disks).

Once I got through those problems, I was able to boot from my vhd and created a nice little testing environment for VS 2010.

2010 Column Planning

hdierking — Fri, 07 Aug 2009 23:55:00 GMT

It’s that time of year once again where I go through my column lineup to see how individual columns are performing and then use that data, along with various other inputs (like what topics I think are emerging and want to get ahead of, etc.) to determine next year’s column lineup. I want to extend my process to you and get your feedback on which columns are working well and which ones aren’t. Additionally, just to give you a bit of additional insight, here is some of the data that I’m looking at as I go through the process. Obviously, this is just one dimension of performance, but it’s also one that has a pretty large sample size so I generally trust it more than I do other measures, such as the “5-star” ratings.

So give me your thoughts. If it were up to you, which columns would you run more frequently? Which ones would you cancel? Or more generally, is there anything on this chart that surprised you?

R# Refactoring Steps For Introducing a Const

hdierking — Tue, 21 Jul 2009 21:03:14 GMT

I’m currently writing some pipeline components for SSIS and I’m curious as to whether there is a commonly followed refactoring pattern for extracting constants. For example, consider the following.

var authenticationType = ComponentMetaData.CustomPropertyCollection.New();
authenticationType.Name = C_AUTHENTICATIONTYPE;

There’s nothing unique to this code. In fact, if you looked further at the surrounding code, you would see a whole bunch of similar declarations – and that’s where it can get a bit annoying. If C_AUTHENTICATIONTYPE were anything other than a const, I wouldn’t think twice about simply typing in the above code and letting ReSharper prompt me to create a variable/field/etc. However, I haven’t found a similar pattern for creating a const – and I *hate* the idea of manually entering the const declaration as it throws me out of my rhythm.

So here’s what I’m currently doing – let me know if there’s a better way.

First, I’m having R# create a new local variable (standard alt+enter). Next I’m executing the “Introduce Field” refactoring on the new variable and selecting the “Introduce constant” option (note that if you haven’t initialized your local variable, this option will be unavailable).

This works pretty well – certainly better than having to context switch in order to go declare a const. I did see that there is a “to const” option for variables. However, this option creates a local const which doesn’t help all that much, and I couldn’t find a secondary refactoring to extract the const from the method to the class scope (at least not in R#).