Poco, Mef, and custom type systems. Are you ready to take the red pill?

The Matrix

http://datacore.sciflicks.com/the_matrix/images/the_matrix_large_08.jpg

I am about to show you in this post an approach to how poco (Plain Old Clr Objects [without attributes]) can be achieved with our current MEF codebase through creation of a new type system.

Are you ready t0 take the red pill?

Before you do, let’s set a bit of context. You may find it surprising to know that in general I am not a huge fan of attributes, at least from the aspect of consumer code. There are several reasons for this, but at the top of my list is that they increase the signal-to-noise ratio of the code, reduce scan-ability, and also couple the code to a specific implementation.

That being said, MEF uses attributes (at least our attributed programming model does) for discovery of parts. Attributes provide a means for part authors to express that a part is part, thus making it discoverable. They provide a means for a host to interrogate the available parts, and make decisions on what to load, as well as adapt dynamically based on that present set. Attributes provide another benefit which is static analyzability meaning that tooling can scan assemblies and automatically determine what is present. It works really well for extensible systems like Visual Studio, where there are 1 to n implementations of extensions provided by an unlimited set of ISVs. We like to refer to this as external extensibility, or implicit composition (composing based on what is present, rather than registration).

But what happens when the set is not open-ended, which is common when you are composing within your application (otherwise known as internal extensibility)? In these cases adding attributes may seem unnecessary as you know the set of parts that you have available. Isn’t there some way to just tell MEF, “Use these!”? Up until now our answer to this question has been that our attributed model does not support this, but in the future custom programming models can be created that sit side-by-side with our attributed model which will support non-attributed configurations.

As a matter of fact, someone has already proven this to be true. Andreas Håkansson has gone and implemented a generic provider based model that allows providing alternative means for defining parts. He’s got some great sample providers that work with his model that allow doing fluent interface configuration, or app.config style. These models introduce a bunch of rich capabilities that our current model does not such as explicit defaults, custom activation, etc. As far as I know, Andreas model is the only current available programming model for non-attributed parts.

Achieving Poco through a custom type system.

Recently we discovered however there is an additional alternative approach to new programming models which we are exploring. That is, build a custom type system :-) This may sound like you’ve stepped into the Matrix, but it’s not as crazy as it seems.  In this case I’d argue though that ignorance is not bliss. ;-)

A few caveats and warnings on this approach

Creating custom derived System.Type implementations is not fool-proof, and there are a bunch of known issues.

For one thing support for custom types is not throughout the framework, there are a few places in the framework which actually depend on RuntimeType, the default concrete implementation of type in the framework. In our case this doesn’t effect us because MEF ‘s attributed model does not have this dependency. If however you tried to pass custom Types throughout the framework you will find that some things might break.

Creating custom type systems allows you to break  a lot of rules that tend to take for granted, similar to stepping into the matrix :-). You can easily have side-effects in your custom type system that could have ripple effects throughout your app. In the cases here, we are additive, and are not changing the existing underlying functionality.

The Red Pill – Type is Abstract

Before we go further, let me let out a small, but apparently not well known secret. System.Type is abstract.

Don’t believe me? Check this screenshot below from Reflector. Actually not only is Type abstract, but ConstructorInfo, PropertyInfo and all the other infos are also abstract.

image

Your probably sitting in shock at this point, and rethinking everything you thought you knew about the type system. Hmm. OK, so once we’ve digested that, we can think whether not it has any implications for MEF? Well MEF has a TypeCatalog which accepts a collection of types. Our attributed model reads through the collection of types and their attributes to build up part definitions of imports, and exports.

If THAT is the case, then conceptually can’t we create a new type (basically an adapter) which will take an underlying non-attributed type, read it’s members and then create attributes for those types based on a set of conventions? It is possible.

Here’s passing unit tests to prove it.

[Test]

public void When_poco_type_is_reflected_then_export_attribute_is_returned()

{

    var mefType = new ComposablePartType<MyPoco, IPoco>();

    var exportAttribute = mefType.GetAttributes<ExportAttribute>().FirstOrDefault();

    Assert.IsNotNull(exportAttribute != null);

}

 

[Test]

public void When_export_attribute_is_returned_then_contract_name_matches()

{

    var mefType = new ComposablePartType<MyPoco, IPoco>();

    var exportAttribute = mefType.GetAttributes<ExportAttribute>().FirstOrDefault();

    Assert.AreEqual(typeof(IPoco), exportAttribute.ContractType);   

}

 

[Test]

public void When_poco_type_is_reflected_then_part_creation_policy_attribute_is_returned()

{

    var mefType = new ComposablePartType<MyPoco, IPoco>(CreationPolicy.Shared);

    var partCreationPolicyAttribute = mefType.GetAttributes<PartCreationPolicyAttribute>().FirstOrDefault();

    Assert.IsNotNull(partCreationPolicyAttribute != null);

}

 

[Test]

public void When_part_creation_policy_attribute_is_returned_then_is_shared()

{

    var mefType = new ComposablePartType<MyPoco, IPoco>(CreationPolicy.Shared);

    var partCreationPolicyAttribute = mefType.GetAttributes<PartCreationPolicyAttribute>().FirstOrDefault();

    Assert.AreEqual(CreationPolicy.Shared, partCreationPolicyAttribute.CreationPolicy);

}

 

 

[Test]

public void When_poco_type_is_reflected_then_importing_constructor_is_returned()

{

    var mefType = new ComposablePartType<MyPoco, IPoco>();

    var constructors =

        mefType.GetConstructors().Where(c => c.GetAttributes<ImportingConstructorAttribute>().Count() == 1);

    Assert.IsTrue(constructors.Count() ==1 );

}

 

[Test]

public void When_poco_type_is_added_to_type_catalog_then_part_is_created()

{

    var catalog = new TypeCatalog(new ComposablePartType<MyPoco, IPoco>(CreationPolicy.Shared));

    Assert.AreEqual(1, catalog.Parts.Count());

}

 

[Test]

public void When_poco_type_is_added_to_type_catalog_then_ipoco_export_can_be_retrieved()

{

    var catalog = new TypeCatalog(new ComposablePartType<MyPoco, IPoco>(CreationPolicy.Shared));

    var part = catalog.Parts.First();

    var export = part.ExportDefinitions.Where(ed => ed.ContractName == AttributedModelServices.GetContractName(typeof(IPoco)));

    Assert.IsNotNull(export);

}

If you are thinking at this point, “Show me the code”, go get it from my skydrive here. In the download you’ll find a new TypeExtensions library along with unit tests. Disclaimer: This is a prototype at this point and is not production code.

Creating a new type

Take a look at the first test. I am creating a new type called ComposablePartType which is generic, and I am passing in MyPoco as the implementation type, and IPoco as the contract type.

public class ComposablePartType<TImplementation, TContract> : TypeDelegator
{}

Notice it inherits from TypeDelegator, well what the heck is that?

image

TypeDelegator is a lesser known member of the framework. It inherits from System.Type, accepts a type which it wraps, and then delegates all calls on itself to. TypeDelegator is really handy, because without it I would have to manually implement and delegate to all members myself

Export and PartCreationPolicy attributes

Below is the code for MyPoco and IPoco, “Look Ma, no attributes”

public class MyPoco : IPoco
{
    public MyPoco(bool isPoco, int theAnswer, ILogger logger)
    {
        this.IsPoco = isPoco;
        this.TheAnswer = theAnswer;
        this.Logger = logger;
    }

    public bool IsPoco { get; internal set; }
    public int TheAnswer { get; internal set; }
    public ILogger Logger { get; internal set; }
}

public interface IPoco
{
    bool IsPoco { get; }
    int TheAnswer { get; }
    ILogger Logger { get; }
}

The unit test is then querying the attributes on this “type” to then see if it can find an ExportAttribute, which assuming the test succeeds it will. ComposablePartType is for all intents and purposes just like System.Type. With one major difference, it creates attributes on the fly based on underlying conventions on the Type it wraps. In this case it has created an Export attribute, exporting the IPoco contract, which is verified in the second test.

Let’s look at the next two tests which relates to part creation policy. The first test is checking whether or not the type has a PartCreationPolicyAttribute it, and the second test is checking that the attribute is set to shared. OK, that’s just an extension pattern wise of the previous tests.

Here’s how it’s done under the hood.

public override object[] GetCustomAttributes(bool inherit)
{
    if (_attributes == null)
    {
        _attributes =  base.GetCustomAttributes(inherit);
        Array.Resize(ref _attributes, _attributes.Length + 2);
        _attributes[_attributes.Length - 2] = 
            new ExportAttribute(_contractName, typeof(TContract));
        _attributes[_attributes.Length - 1] = 
            new PartCreationPolicyAttribute(_partCreationPolicy);
    }
    return _attributes;
}

Above, I am taking the collection of attributes returned from the underlying type, increasing the size of the collection and adding the Export and PartCreationPolicy attributes. Once i do this, the type has become attributed ;-)

Greedy Constructors and parameter name conventions

Allright, let’s peel back the onion a little bit further and look at the next test, this one relates to constructors. In the test, I am querying for a constructor to see if I can find one that has an [ImportingConstructor] attribute. Notice that MyPoco has a single constructor and it is not parameterless. The new type system supports finding the greediest constructor. It finds that constructor and then adds an [ImportingConstructor] on it in order to make MEF’s attributed model happy.

I have to say, I just adore Linq. Look at this code which handles finding the greediest constructor.

public override ConstructorInfo[] GetConstructors(BindingFlags bindingAttr)
{
    if (_constructors == null)
    {
        _constructors =
        base.GetConstructors(bindingAttr).OrderByDescending(
        c => c.GetParameters().Count()).ToArray();
        _constructors[0] = new ComposablePartConstructorInfo(
            _constructors[0],_alwaysUseConstructorParamNameAsContract);
    }
    return _constructors;
}

Once I have the greediest constructor, I replace it with a new custom ComposablePartConstructorInfo which wraps the existing constructor.

public class ComposablePartConstructorInfo : ConstructorInfo {}

_alwaysUseConstructorParamNameAsContract specifies whether or not the imports on the params for that constructor should be the parameter name or the contract. In MEF today, we default to use the type. Having this level of control at the type level allows me to augment that with additional conventions.

ComposablePartConstructorInfo looks at the parameters of the constructor. If they are primitives, or types String/DateTime, or if the override params is set,  it will create a ComposablePartParameterInfo which in turn adds an [Import] attribute specifying the parameter name as the contract. This is useful for cases where you have a sender parameter of type string where the contract is “Sender”.

public override ParameterInfo[] GetParameters()
{
    if (_parameterInfos == null)
    {
        List<ParameterInfo> parameters = new List<ParameterInfo>();
        foreach (ParameterInfo param in _constructorInfo.GetParameters())
        {
            if (_alwaysUseParamNameAsContract || 
                param.ParameterType.IsPrimitive ||
                param.ParameterType == typeof (string) || 
                param.ParameterType == typeof (DateTime))
                parameters.Add(new ComposablePartParameterInfo(param));
            else
                parameters.Add(param);
        }
        _parameterInfos = parameters.ToArray();
    }
    return _parameterInfos;
}

By now, I think you’re starting to get the picture of how this all fits together. I’ll leave you to explore ComposablePartParameterInfo yourself ;-)

I’ll leave you with one other thought, you can take ComposablePartConstructorInfo  even further to do AOP-style interception and dynamic proxying. Can you see how?

Registering poco types with Part Registries

Now that we have our custom type system, we can start to feed these types to MEF. One approach to doing this is to simply use our TypeCatalog.  Combined with Linq you can do some pretty nifty things once you have an assembly in hand like registering all types that end in Service, or all types that live in the MyApp.Services namespace. That’s all well and good, but the one thing you lose in doing so is the reflect-ability and static-analysis that our attributed model currently provides, as these parts are registered on the fly at runtime. It would be nice if there was a way to combine the two approaches. There is, ComposablePartRegistry and PartRegistryCatalog.

A registry is a class that contains a catalog. Catalogs in MEF contain parts. ComposablePartRegistry is below.

public abstract class ComposablePartRegistry
{
    protected abstract ComposablePartCatalog GetCatalog();
    public ComposablePartCatalog Catalog
    {
        get { return GetCatalog(); }
    }
}

To register a set of pocos, you simply derive from ComposablePartRegistry and implement the GetCatalog method to return a TypeCatalog.

public class TestRegistry : ComposablePartRegistry
{
    protected override ComposablePartCatalog GetCatalog()
    {
        List<Type> types = new List<Type>();
        types.Add(new ComposablePartType<MyPoco, IPoco>
            (CreationPolicy.Shared));
        types.Add(new ComposablePartType<Logger, ILogger>
            (CreationPolicy.Shared));
        return new TypeCatalog(types);
    }
}

Above TestRegistry is registering two parts, MyPoco and Logger both as shared. The last piece of the puzzle is to make registries discoverable. Once they are discoverable we can get back our full analyzability. This is where PartRegistryCatalog comes in.  This new catalog inherits from our AssemblyCatalog and adds in additional discovery of part registries. The important part is in the method below.

private void LoadRegistries()
{
    List<ComposablePartDefinition> parts = 
        new List<ComposablePartDefinition>();
    var registries = new List<ComposablePartRegistry>();

    var registryTypes = Assembly.GetTypes().
        Where(t => t.IsSubclassOf(typeof(ComposablePartRegistry)));

    foreach (var registryType in registryTypes)
    {
        var registry = (ComposablePartRegistry)Activator.
            CreateInstance(registryType);
        registries.Add(registry);
        parts.AddRange(registry.Catalog.Parts);
    }
    _parts = parts.AsQueryable();
    this.Registries = registries;
}

The method grabs all registries that it finds and then instantiates them. Once the registry is created, it’s parts are queried and stored within the registry catalog. When the catalog’s Part property is invoked, it will return a combination of parts discovered through registries and those discovered through the normal means.

Bringing it all together

Now with all the pieces in place, we can start composing pocos in the container. First we setup our container. Our current assembly has TestRegistry which will be discovered by the registry catalog. Next we can add our values to the container so that they can be injected.

public CompositionContainer SetupContainerUsingConstructorParamTypes...()
{
    var catalog = new 
        PartRegistryCatalog(typeof(ComposablePartTypeFixture).Assembly);
    var container = new CompositionContainer(catalog);

    //forcefully add some values to the container. 
    _isPoco = true;
    _theAnswer = 42;
    container.ComposeExportedObject("IsPoco",_isPoco);
    container.ComposeExportedObject("TheAnswer",_theAnswer);
    return container;
}

Finally we can run our tests which should be self-explanatory.

[Test]
public void When_..._queried_from_the_container_then_Wow_is_injected...() 
{
    var container = 
        SetupContainerUsingConstructorParamTypesAsTheContractName();
    var poco = container.GetExportedObject<IPoco>();
    Assert.AreEqual(_isPoco, poco.IsPoco);
}

[Test]
public void When_..._queried_from_the_container_then_42_is_injected...()
{
    var container = 
        SetupContainerUsingConstructorParamTypesAsTheContractName();
    var poco = container.GetExportedObject<IPoco>();
    Assert.AreEqual(_theAnswer, poco.TheAnswer);
}

Summing up and what’s next?

At this point this is just a prototype illustrating a modified type system which dynamically creates attributes on non-attributed part through a set of conventions. This approach is very powerful in that it allows managing a closed set of parts, such as the internals of an application, and works harmoniously within our existing attributed programming model.

A few differences / limitations of the current prototype.

1. The Type is the export and there is only one contract exported. The model does not currently support exporting methods and properties. It could, it just doesn’t. This was deliberate in order to keep the model simplified.

2. No metadata support. Currently ComposablePartType does not provide any means for providing metadata for the part. This was also to keep things simple, though it is possible to support it.

3. Recomposition is not supported. Currently all imports that are created are not recomposable.

We’d like your feedback on the approach in order to let us know whether we should take it further, or even consider putting it into Mef V1.

Let us know!

Download the code here

This entry was posted in MEF, poco. Bookmark the permalink. Follow any comments here with the RSS feed for this post.
  • http://wizardsofsmart.net/ Ryan Riley

    I love it, at least the theory. I’ll have to have a play to grok it better, but this is really cool. So many possibilities seem so open. One thing that I liked about attributed-MEF that I didn’t see here was the ability to auto-register contracts from assemblies. Maybe there’s another convention that could be used here (folder for contracts/interfaces and another for types?), but we’ve tried hard to keep the noise of manual registration down in our apps. I can’t imagine that would be hard to do.

    As for your questions, I may just be too young in the tooth with containers to understand, but I’ve never used 1, I’m not sure what you mean by 2 (we can’t attribute it in any other way and have it retained, e.g. Serializable?), and I don’t understand what you mean by not recomposable in question 3. Can you elaborate more?

  • http://codebetter.com/members/gblock/default.aspx Glenn Block

    @Tim, TypeDescriptionProvider is an opton however, it only works if you access the information through a specific set of APIs, which our attributed model does not use.

    A further option is having a class that implements ICustomAttributeProvider which is actually what MEF looks for under the hood, however it would require a bunch of custom work and would not work with our existing catalogs.

    The goal of this “spike” was to provide an adaption mechanism that worked with all of our existing MEF infrastructure.

  • Tim Lovell-Smith

    This seems extremely hacky – the way the custom types generated will only attach a single attribute in each custom type generated and there is no ‘common fake type’ that all attributes for a particular ‘real type’ can attach to (when multiple components want to modify the same type).

    Aren’t there already established ways of attaching attributes after the fact like using e.g. TypeDescriptionProvider from System.ComponentModel, or MetadataStore from Windows.Design

  • http://codebetter.com/members/gblock/default.aspx Glenn Block

    Hi Joe

    Thanks for the kind words on the team.

    As far as using an IoC, I wouldn’t say you have to choose between an IoC and MEF. We are trying to make MEF more palatable in terms of removing attributes for internal composition, but there are still significant differences. Each IoC container offers different things that make it special, we do NOT want to force people to give that up.

    We are working on a path to how IoC containers could integrate cleanly with MEF’s underlying part model therefore having a single container in the app. We are working with p&p for example on MEF-enabling Unity. We are also working with a bunch of the IoC authors including Hammett, Nate, Jeremy, et al to get similar support added to their containers.

    You can see a preview of what this might look like on Nick Blumhardt’s blog post here http://blogs.msdn.com/nblumhardt/archive/2009/03/16/hosting-mef-extensions-in-an-ioc-container.aspx, or in his MEF integration library for Autofac which you can get on the Autofac site here http://code.google.com/p/autofac/wiki/MefIntegration,

    Appreciate your thoughts.

  • http://blogs.windowsclient.net/joeyw Joe

    Thanks for the responses Glenn. I think it’s great work that you and your team are doing.

    My problem is that we have an in-house IoC and I want to be ready for .NET 4 and adopt MEF. At the same time I’m looking for guidance of how MEF and PRISM can work together. This is long-term planning, rather than immediate solutions – so it’s a matter of trying to second guess the roadmap for MEF further down the road.

  • http://www.twitter.com/talktomikesmith Michael A. Smith

    @Glenn Thanks for responding. Oh, I absolutely agree, the benefits definitely outweigh the risks, and it should be prioritised as such. As you said, full AOP support would be terrific.

    On a side note, with respect to potential MEF features, is your team encountering any that produce the sort of risk whereas they won’t be able to ship in time?

  • http://codebetter.com/members/gblock/default.aspx Glenn Block

    @Joe, here’s a simple model from one of our devs that can do just that without requiring a custom type. Our new APIs do make it far simpler :-)

    http://pastie.org/460259

    Is this along the lines of what you are thinking?

    The one thing it does not cover is AOP, however a cut down version of ComposablePartType / ComposablePartConstructorInfo can be introduced to allow adding custom activation hooks.

  • http://codebetter.com/members/gblock/default.aspx Glenn Block

    @Joe that underlying API is our primitives (ComposablePartDefinition, ImportDefinition, ExportDefintion), none of these know anything about attributes. This meansIoC containers can use and integrate with them, but they would need to do the work to do so. The question is when/where that makes sense.

    Nick Blumhardt has a good post on this on his blog here.

    http://blogs.msdn.com/nblumhardt/archive/2009/03/16/hosting-mef-extensions-in-an-ioc-container.aspx

  • http://blogs.windowsclient.net/joeyw Joe

    Glenn, I see your point that is additive. In that case, if it’s replaced with something more strategic later there’s no real damage done.

    The reason why I say hacky is that MEF depends on the type system, so we simulate attributes using a new type system on top of the existing one. OK, easy enough – but as you allude to – attributes should be just one way of registering types. Maybe there just needs to be some underlying API that some existing IoC libraries can use if they wanted to leverage MEF’s dependency resolution engine.

  • http://home.infusionblogs.com/kszklenski/default.aspx Kyle Szklenski

    The test error with IsNotNull is in your first test.

  • http://codebetter.com/members/gblock/default.aspx Glenn Block

    @Leonid, why?

    @Paul thanks.

  • http://codebetter.com/members/gblock/default.aspx Glenn Block

    @Michael

    Thanks for reviewing and finding my bugs :-), where exacty is it, which test?

    Thanks
    Glenn

  • http://codebetter.com/members/gblock/default.aspx Glenn Block

    @Rob I hope it didn’t splatter too much, we still need you for Caliburn :-)

  • http://codebetter.com/members/gblock/default.aspx Glenn Block

    @Joe.

    What about it feels hacky? It is basically an adapter which layers on top of a non-attributed type.

    MEF actually is not bound to attributes, our attributed programming model is. However, today we do not have a non-attributed model yet. This is a good feedback though.

    Actually as I am writing this, one of the devs on my team just emailed me a very simple programming model for doing just that on our new ReflectionModel abstractions in preview 5. You may get what you are asking for. ;-)

  • http://codebetter.com/members/gblock/default.aspx Glenn Block

    Hi Michael

    Thanks for the feedback. Yes there are strategies for IoC containers to work with MEF, and we will continue to develop these further. At a minimum it will require work on each of the IoC container authors in order to support MEF as one of their component sources.

    On the other hand there are a large set of folks that want to use MEF internally within their apps that do not currently today use any of the existing IoC containers. The model proposed here allows them to do so, without incurring the maintainability costs of additional attributes.

    Granted it does introduce ambiguity. Personally I think the benefits outweigh the risks.

    What do you think?

  • http://www.vcskicks.com Visual C# Kicks

    Creative solution, really makes you think

  • http://adventuresinsoftware.com/blog Michael L Perry

    Maybe it’s just the trip down the rabbit hole, but it looks like your use of IsNotNull is incorrect. It expects an object reference, not a condition. The way you are using it, you’ll always pass a boxed boolean, which would of course not be null.

  • http://www.bluespire.com Rob

    My brain just ‘sploded and a whole lota’ awesome was splattered everywhere!

  • http://www.paulbatum.com Paul Batum

    Great post Glenn, I had no idea that Type and the xInfo classes were abstract!

    My mind is awash with half-possibilities. Awesome stuff.

  • http://blogs.windowsclient.net/joeyw Joe

    I’m not sure – this just feels a bit hacky to me.
    Surely the right way to go would be to have a type registration catalog, with attributes as just one way to feed into it. Just like an IoC Container.

  • http://blog.shirmanov.com Leonid Shirmanov

    I believe it absolutely should be in MEF 1.0

  • http://www.twitter.com/talktomikesmith Michael A. Smith

    Unless, I’m overshooting, I see the “new type system” approach as a beginning to being able to augment MEF with capabilities normally reserved for IoC containers.

    There’s always something to be said about not re-inventing the wheel, and given that strategies exist for IoC containers and MEF to interop well, clearly this feature is not crucial. Nevertheless, there will be folks who will feel more comfortable using MEF in this manner in conjunction with the attribute programming model, and avoiding the additional complexity of adapting an IoC container.

    As I’m still not fully across the bounds of using a custom Type system across the .NET Framework, it is hard for me to weigh this approach up against the Provider Model in MEF Contrib. Either way, I think some approach for supporting POCO’s like this should be core. Give us our cake and we’ll decide whether or not to eat it (please).

  • http://www.twitter.com/talktomikesmith Michael A. Smith

    Very cool. To offer the feedback you are looking for, I”ll have to consider it some more. Just might add that the paragraph that goes “TypeDelegator is a lesser known… to all members myself” took me a few times to read to grok. Thanks as always, Glenn.