Karl Seguin

NHibernate : Handling the Special Cases

karl — Thu, 18 Jun 2009 14:03:00 GMT

In my last NHibernate post I mentioned that NHibernate is great because it does the heavy lifting for the common tasks that make up 95% of my data access layer, while letting me tweak the remaining 5% as needed.

Let's look at some of the ways you can tweak, or even circumvent NHibernate for those queries that require some extra attention. Do note that there are a lot of different solutions – we're only looking at a few.

The most obvious method is to use named queries and custom SQL statements in your mapping. This provides you a lot of flexibility without diverging from NHibernate's realm. The NHibernate reference has good documentation on this approach, so we won't cover it (check out section 14.2 "Named Queries", 14.3 "Custom SQL for create, update and delete" and 14.4 "Custom SQL for loading").

The next option is to use an ISqlQuery via the ISession.CreateSqlQuery. Unlike an IQuery which works against your domain using HQL, an ISqlQuery is closely tied to your database and SQL. However, you still get access to NHibernate parameter substitutions, paging capabilities and O/R mapping (if you want). Again, the NHibernate reference has a very detailed section, check out 14.1 "Using an ISQLQuery".

You can optimize your Deletes by supplying the ISession.Delete command some HQL rather than the instance you want to delete. For example, if you want to delete all comments belonging to a blog post, you can do:

session.Delete("from Comment where PostId = :id", postId, NHibernateUtil.Int32);

More recently the NHibernate team made it possible to execute any statement (insert/delete/update) from an IQuery. You can find out more from this post by Ayende.

When necessary (typically bulk imports or other bulk operations), you can circumvent NHibernate's first-level cache by using an IStatelessSession rather than an ISession. Rather than calling OpenSession on your ISessionFactory, simply call OpenStatlessSession. Keep in mind that the IStatelessSession doesn't have the exact same API as an ISession.

When things get really crazy, you can always get an ADO.NET connection from the ISession's Connection property. With a raw database connection, the world is your oyster. You can always write your code in such a way that you have direct access to our connection string to create your own connection, thus completely and fully circumventing NHibernate.

Finally, any of these methods can be combined with some simple designs in order to ensure that everything is neat and tidy:

public abstract class Repository : IRepository
{
    public void Delete<T>(T entity)
    {
        WithinTransaction(s => s.Delete(target));
    }

    protected void WithinTransaction(Action action)
    {
        WithinTransaction(Session, action);
    }        
    protected static void WithinTransaction(ISession session, Action action)
    {
        var transaction = session.BeginTransaction();
        try
        {
            action();
            transaction.Commit();
        }
        catch (Exception)
        {
            transaction.Rollback();
            throw;
        }
        finally
        {
            transaction.Dispose();
        }
    }
}


public class PostRepository : Repositor, IPostRepository
{
    public void Delete(int postId)
    {
        WithinTransaction(s => 
            {
                session.Delete("from Comment where PostId = :id", postId, NHibernateUtil.Int32);
                session.Delete("from Post where Id = :id", postId, NHibernateUtil.Int32);
            });
    }
}

Once you understand NHibernate's flexibility and understand your code's hot spots, it generally takes a pretty trivial amount of tweaking to speed things up. While the changes you make, in the name of performance, to your models (both domain and data) may be disgusting rest assured that HHibernate will not stand in your way. Be warned though that many of the above solutions may have side effects with respect to NHibernate's 1st and 2nd level caches. However, if you were to custom write everything, including caching, you'd likely have similar synch issues.

NHibernate : Some Naked Thoughts

karl — Thu, 11 Jun 2009 01:07:00 GMT

Please, Just Remember One Thing

Just for a few seconds forget everything you know about NHibernate.

Good.

Now, before I snap my fingers and everything comes flooding back, the only thing you need to know about NHibernate is that whenever you need to, you can easily circumvent it and do whatever is needed to get the job done (like use an IDbConnection and IDbCommand to execute a stored procedure).

**snap**

I often hear people saying it, and often say it myself, but people just don't seem to get it. You use NHibernate to save yourself from having to write a ton of code for 95% of your data access, and then, after profiling, optimize the critical 5% with whatever other approach you prefer. Its stupidly easy. You can access an IDbConnection directly from an NHibernate's ISession, or subclass your NHibernateDataAccess class with an SqlServerDataAccess class and override those critical sections.

Instead of taking the above sane approach, I see people laboring through huge amounts of mundane code in order to keep control over that unique 5% (maybe a special report, or performance sensitive path). I see people concerned about performance, without understanding that for most of what an application does, the performance difference between method A, B and C are indistinguishable, whereas the maintenance and developer costs vary hugely.

Essentially I see people who are worried about not being able to tweak when necessary and people who simply don't want to change.

To the first group I say that your fears are understood, but not to worry. You can always drop down to the assembly of data access (SQL, Sprocs, ...).

Four Areas of Improvement

NHibernate makes my life easier and my code better, but it isn't without problems. A lot of these have to do with the fact that it's trying to solve a complex problem, of which numerous variations exist. As such, NHibernate's biggest problem is that it's hard to learn and harder still to master. There are some specific pain points I'd like to see addressed.

First, mapping needs to be simplified. It's extremely intimidating to a newcomer and requires that you understand most of the fundamentals. This is already being addressed with FluentNhibernate and hopefully the project will continue to mature.

Secondly, collections and references can be particularly puzzling. From the names used (bags, maps and sets are all foreign to most .NET developers), to advanced features like caching, lazy loading and cascading. I'm not sure if this is something that can actually be improved, or if this battle needs to be fought via documentation.

Thirdly, neither HQL nor Criterias feel like the most natural querying languages. Hopefully development on Linq to NHibernate will continue to be a priority.

Finally, NHibernate is an abstraction that will leak. This shouldn't be unexpected, but it would be nice to try and address some of the worst offenders. The leaks that continuously trip me up are related to management of ISessions. Lazy loading and session management is a black art – doubtlessly easy for masters, and frustrating for everyone else.

NHibernate and Microsoft

Microsoft has long been criticized for its attitude towards open source frameworks. The common approach appears to be to compete with established open source frameworks using inferior solutions. MSTest, MSBuild, Sandcastle, ASP.NET Ajax, ASP.NET MVC, Entity Framework, Unity, Logging Application Block, Velocity, Team Foundation SSC are all examples that come to mind.

It'd be nice if Microsoft took a different approach with NHibernate, by not only acknowledging its presence and using it in samples and demos, but also actively contributing to it. Remember, during NHibernate's lifetime, Microsoft has invented three distinct data access strategies (DataSets, Linq to SQL and the Entity Framework). So much effort, so much relearning by developers, to end up with a poor replica of NHibernate years later.

A couple years ago, Google contributed Hibernate Shards to the Hibernate project (and I'd bet that they've been involved in Hibernate in other ways since then). Surely Microsoft can help fund the project in a manner that works with the NHibernate team (money or resources).

NHibernate and You

I do think it's important for any developer to learn new things. Equally important is to learn them from different people. Its important that you learn Ruby or Python, not because you should use them, but because seeing MVC over ActiveRecord completely changes your perspective on how applications can be built. The more approaches that you know, the more likely you are to pick the right one for the job. If you only know one approach then you'll always use that one – and you better hope like hell that the person selling you that approach knows what he or she is talking about.

As for NHibernate specifically, most developers should learn about O/R mappers. You should know the advantages and disadvantages and therefore know when a situations calls for it or not.

Revisiting CodeBetter.Canvas

karl — Mon, 25 May 2009 13:02:00 GMT

A month ago I released CodeBetter.Canvas, a simple application with equally simple goals:

Provide developers with a starting point for new ASP.NET MVC projects
Provide developers with a learning tool for oft-talked about tools/patterns

Instead of building a full-fledge application (like another blog engine, or shopping site), I wanted to build something as close to a blank canvas as possible - not only because it made the task more managable for me, but also because I thought it would be a better way to learn, and a more practical application for developers to use as a launching pad for their own real projects. While being an MVC application, the focus is more general: namely proper use of dependency injection, leveraging NHibernate/FluentNhibernate/Linq2Nhibernate, unit testing and mocking, and jQuery.

Yesterday I released an updated version of CodeBetter.Canvas to Google Code. You can check it out at http://code.google.com/p/codebettercanvas/. There are a number of new things in this release, including a revamped repository model, the introduction of a model binding framework (these end up doing a lot of heavy lifting), an improved validation framework, and ideas on how to do paged lists.

Enjoy.

2 years ago I gave up on Silverlight

karl — Sat, 23 May 2009 15:51:00 GMT

Almost two years ago, to the day, I completely gave up on Silverlight. How do I remember the day so clearly? Because every now and I again, I get an email notification that someone has replied to my following forum post: http://silverlight.net/forums/p/1334/216456.aspx. In it, I asked why the Shape classes are sealed. At the time I figured it might have just been a early-release kinda thing. However, since the issue continues to be raised I'll assume the problem still hasn't been fixed.

This whole thing reminds me of another common .NET problem - internal framework classes. This was a hot topic with the release of .NET 3.5 and the ExpressionVisitor code, so much so that a David Kean from Microsoft jumped in to explain the situation. For the most part though, all it did was highlight how inefficient and unwilling to change Microsoft was. Why are classes internal? Because the process to make them public involves 20 some people and a multi-layered review system. There's no "how do you guys handle these problems?" or "where can we improve" - merely a statement of fact of how things are (and therefore must be).

Back to the sealed silverlight classes. This to is a pretty old topic - remember Java is virtual by default, C# is not. What annoys me though, is the approach Microsoft has taken. I asked Why are the classes sealed, and the official response was Why do you want them unsealed? If .NET is built on the basis that developers need to justify how they use the framework, we're in deep shit. The default behavior of a framework developer should be to open components, and only close them off when necessary. Otherwise it limits what your customers can do with your product, places assumptions on how your framework will be used, and makes it clear that you think you know better than we do. Lets not forget that coming up with a use-case for inheriting Line isn't particularly difficult - maybe I want to create my own DottedLine class?

The specific wording used by Nick Kramer (Silverlight's Program Manager) was:

"We want to err on the side of less API surface area -- if you're missing something important someone will tell you, but if you include a feature that doesn't get used you'll never hear about it, and will pay the download cost for it forever. We weren't able to think of a compelling reason for arbitrary subclassing, but maybe we weren't creative enough -- what scenarios do you have in mind?"

It isn't Nick Kramer & Team's job to think of reasons for subclassing (its our job to do that). Nick Kramer and Team should only be concerned with reasons not to have an unsealed class. Nick did provide some of that reasoning, but I'm unconvinced. Essentially coming down to performance and wrapping unmanaged objects - issues simple enough to cover via documentation.

Making the untestable testable with Anonymous Methods and Dependency Injection

karl — Fri, 08 May 2009 14:42:00 GMT

Making the untestable testable with Anonymous Methods and Dependency Injection

It can be frustrating to want to write unit tests, only to hit some code which is rather untestable. Take for example, the following code:

public class MailSender
{
   public void SendActivationMail(User user, string activationCode)
   {
            string body = GetCopyOfActivationBody(); //implementation doesn't matter
            body = body.Replace("#NAME#", user.Name).Replace("#CODE#", activationKey).Replace("#EMAIL#", user.Email);
            var message = new MailMessage("support@piccolo.cool", user.Email, "Account Activation", content) {IsBodyHtml = true};
            new SmtpClient().Send(message);
   }
}

Obviously, I can't test this method as-is without making sure that a mail server is setup. And even if it was, how would I verify that the properties of my MailMessage object were properly set?

In the past, I probably would have solved this by using a virtual method and mocking it:

public void SendActivationMail(User user, string activationCode)
{
        string body = GetCopyOfActivationBody(); //implementation doesn't matter
        body = body.Replace("#NAME#", user.Name).Replace("#CODE#", activationKey).Replace("#EMAIL#", user.Email);
        var message = new MailMessage("support@piccolo.cool", user.Email, "Account Activation", content) {IsBodyHtml = true};
        SendMessage(message);
}
public virtual void SendMessage(MailMessage message)
{
   new SmtpClient().Send(message);
}

Now we can create a mock of our MailSender class and mock our SendMessage method. I still think this is an acceptable solution, but I now prefer to inject an anonymous method as a dependency. The benefit is that my unit tests are more straightforward, and I'm not having to expose a method like SendMessage.

Before you move on, you should be familiar with anonymous methods, I've blogged about them in the past and/or you can check out this dimecast.

First we'll change our MailSender class to accept a simple anonymous method into its constructor:

public class MailSender
{
   private readonly Action<MailMessage> _sendMessage;
   public MailSender(Action<MailMessage> sendMessage)
   {
      _sendMessage = sendMessage;
   }
}

Next we'll change our SendActivationMail to use the sendMessage action:

public void SendActivationMail(User user, string activationCode)
{
   string body = GetCopyOfActivationBody(); //implementation doesn't matter
   body = body.Replace("#NAME#", user.Name).Replace("#CODE#", activationKey).Replace("#EMAIL#", user.Email);
   var message = new MailMessage("support@piccolo.cool", user.Email, "Account Activation", content) {IsBodyHtml = true};
   _sendMessage(message);
}

Before we write our test, let's configure this dependency for our normal, production use. Here I'm using Ninject (some version of the 2.0 branch, so the syntax might be slightly different for you):

Bind<Action<MailMessage>>().ToConstant(message => new SmtpClient().Send(message));

That's all there is to it. Now, whenever we get an instance of our MailSender class through Ninject, it'll automatically inject the default behaviour and happily send emails away (for those curious where/how to configure the host/port/username/password, I prefer to do that in the web.config/app.config via the <system.net> section.) As for our tests, they look something like:

[Fact]
public void SendsActivationEmailToUser()
{
    var user = new User {Email = "nail@namekian.cool"};
    var action = new Action<MailMessage>(m => Assert.Equal(user.Email, m.To.ToString()));
    new MailSender(action).SendActivationMail(user, null);
}

There you have it. By injecting our own test-specific anonymous method into our MailSender class we've turned otherwise untestable code into code which is easy to test.

Part of your job should be to learn

karl — Thu, 07 May 2009 12:11:00 GMT

Jan recently asked:

"Something I don't see being addressed that often - When you say you read a lot, is that on your own time, or do you actually spend time at work reading? I think spending time at work reading & learning is justifiable, although not always easy."

My answer is an absolute and irrevocable ABSOLUTELY. You should spend as much time as possible learning while at work, and you should use your work itself as a learning platform. Even in an organization that's constantly under intense client deadlines, you should be able to fit in reading a couple blog posts at the start of your day and during lunch time. It helps if you have a specific list of high-value blogs (as opposed to having to wade through sites like weblogs.asp.net for example). Jeremy Miller recently mentioned that he likes http://www.infoq.com/, so I'm going to start keeping an eye on it. There's also my own aggregate at http://www.foundationsof.com/ which is good if you like codebetter-like content.

Also, I find that there are always some throw-away projects on the go which are ideally suited for experimentation (like that one data import that needs to get done). At the very least, these projects can help you get over some of the technical hurdles of learning a new tool.

You can even find the time to create and maintain your own OSS project at work. For example, I had to write a JSON parser for the compact framework (at the time, none existed). I talked to my manager and, since our company isn't in the market of selling software (or making software for that matter), got approval to release it on CodePlex. In a previous job, I had to write a AMF serializer/deserializer, and since it was based on a GPLd Java Port I also got approval to release it.

Learning at work is something you should talk to your manager about - whether that be for formal training, or simply reading blogs and contributing to OSS projects. Furthermore, the process can be formalized as a team activity, through Lunch and Learns or code reviews. Are you considering using a DI framework, unit testing, or NHibernate at work? Why not download CodeBetter.Canvas and get your colleagues together for an hour to review specific parts of the code?

2009 - A Year of Learning (so far)

karl — Mon, 04 May 2009 14:57:00 GMT

I spend a lot of time learning. I read a lot, work on small projects that let me experiment, and blog (which is probably the best way to learn). I don't see how you can be effective or successful in the long run without a serious commitment to learning. Uncle Bob says it well:

"I think that maintaining your technical chops is a full time job. For that reason I have avoided becoming a business wonk. I hire people to do that for me so I can keep my technical skills as sharp as possible and remain relevant to my profession. I don’t believe I can offer technical advice unless I am living that technical advice."

What I find most interesting is that I'm becoming a better learner. I've always been very eager to improve, but not always effective at it. I'm much better now at filtering through the high noise to signal ratio. A lot of the big sites/blogs I followed years ago are now completely off my radar - too much marketing, too much self promotion, too little focus on craftsmanship and quality. I've also found twitter to be an indispensable tool for picking up on new trends and to see what others are working on/with.

Off the top of my head, here are some of the things I've learnt recently (mix of design and tools):

I've refined my knowledge of DI and am slowly becoming a Ninject ninja,
I finally feel like I'm writing effective unit tests (that was a long road), and I'm hooked on XUnit,
I understand the difference between mocks and stubs, dynamics and stricts, and mock less code but do so more effectively,
I'm starting my 3rd project with NHibernate and (finally) feel comfortable enough to blog/present on it,
I've using FluentNhibernate for a new project, and leveraging NHProf (as of last Friday =] ),
I'm starting my 2nd full ASP.NET MVC project, it's amazing how much cleaner things are now, thanks to past failures (by me and others) and better tooling,
I continue to build small projects in PHP,
I'm slowly learning Python,
I'm using a Mac and making sure I don't forget how to configure apache, use vi and write little bash crons
I have Android on my radar

What interesting tools/design/framework are you learning?

Presenting CodeBetter.Canvas

karl — Wed, 29 Apr 2009 02:25:00 GMT

UPDATE: A new version is available, read more at http://codebetter.com/blogs/karlseguin/archive/2009/05/25/revisiting-codebetter-canvas.aspx

I recently wrote a code-heavy 3-part series on validation. Half-way through, I realized that a sample application would probably help bring together all the incoherent code snippets I was throwing around.

Historically, I've been pretty harsh on shitty code-samples (you can go back to when I first started blogging and probably find some pretty harsh criticism of Adobe and MSDN). As a consequence, I've always tried to write above-average code samples, Of course, more recently I've taken a number of high profile projects to task, namely Oxite and Kobe. As soon as I started putting this sample together, I saw an opportunity to go beyond demo-ing a custom validation library and provide some of the guidance I felt other projects were lacking.

Functionality-wise, all you can do is register a user. The downside of the simplification is that it doesn't address many (most?) of the real challenges you're likely to face. Even I think the entire thing seems rather ivory tower. On the upside you can use this project as a canvas for your own project, or as a launch-pad to learn about all types of exciting technologies. Specifically, despite the extreme lightweightedness, this project makes use of Dependency Injection (via Ninject), OR Mapping (via FluentNHibernate), Unit Testing (via XUnit), mocking (via RhinoMocks) and a couple more trendy topics.

This isn't a perfect example, but I'm pretty happy with it. I think developers who have a genuine interest in learning and enthusiasm for programming will gain a lot from this. I'm sorry for the total lack of documentation, but this is really a help-yourself kinda thing.

I should point out that the way I handle the database path (I use sqlite) is horrible. I did it that way so that the application will [hopefully] work out of the box (all you need in ASP.NET MVC installed). The hack in question is located in the Configuration class. Also, this isn't an opinionated MVC implementation. This isn't really about MVC at all.

If you want help or have a suggestion/comment post a comment here. I'd love to help any way I can, or even better, learn to from my mistakes so that I don't do them again.

Finally, I apologize to the people who just wanted to peak at the validation framework. My demo turned into something more. If I did a good job, you should be able to easily find all of the pieces.

You can download it here

Validation - Part 3 - Server-Side

karl — Wed, 29 Apr 2009 00:08:00 GMT

Validation - Part 3 - Server-Side

<Update>
You can download the sample application from here. Sorry, it turned into more than just a demo of the validation stuff.
</Update>

So far we've built-up a small foundation for a custom validation framework in part 1, and then tied that to a jQuery plugin for our client-side validation in part 2. In this part we'll look at doing server-side validation. Of the three parts, this is the simplest.

Validator

Most of the heavy lifting takes place in our Validator class. Before looking at it, we'll make use of a simple ValidationError object to hold any errors:

public class ValidationError
{
    public string Field { get; private set; }
    public string Message { get; private set; }

    public ValidationError(string field)
    {
        Field = field;
    }

    public ValidationError(string field, string message)
    {
        Field = field;
        Message = message;
    }
}

And here's the powerful Validator:

public interface IValidator
{
    bool Validate(object o, out ValidationError[] errors);
}

public class Validator : IValidator
{
    private readonly IRepository _repository;
    public Validator(IRepository repository)
    {
        _repository = repository;
    }
    
    public bool Validate(object o, out ValidationError[] e)
    {
        var errors = new List<ValidationError>(10);
        var propertyErrors = ValidateProperties(o);
        if (propertyErrors != null)
        {
            errors.AddRange(propertyErrors);
        }
        var customErrors = ValidateCustom(o);
        if (customErrors != null)
        {
            errors.AddRange(customErrors);
        }           
        e = errors.ToArray();
        return e.Length == 0;            
    }        

    private static ICollection<ValidationError> ValidateProperties(object o)
    {                        
        //todo
    }
    private ICollection<ValidationError> ValidateCustom(object o)
    {
        //todo
    }
}

All the Validate method does is take an object and merge any attribute-based errors with any custom-based errors. Here's the implementation for ValidateProperties:

private static ICollection<ValidationError> ValidateProperties(object o)
{                        
    if (!ValidatorConfiguration.Rules.ContainsKey(o.GetType()))
    {
        return null;
    }
    var errors = new List<ValidationError>(10);
    foreach (var property in ValidatorConfiguration.Rules[o.GetType()].Properties)
    {
        var value = property.Property.GetValue(o, null);
        foreach (var validator in property.Validators)
        {
            if (validator.IsValid(o, value)) { continue; }
            errors.Add(new ValidationError(property.Property.Name));
            break;
        }
    }
    return errors;
}

We loop through each validation attribute of each property of our object (assuming any validation rules are configured for it). Notice that we don't add a message to our ValidationError - that's because those are already rendered on the client in the form of the "tip" - we'll get back to that in a bit.

The ValidateCustom is even simpler:

private ICollection<ValidationError> ValidateCustom(object o)
{
    if (!(o is IValidate))
    {
        return null;
    }
    return ((IValidate) o).Validate(_repository);            
}

Here's an example of what a custom Validate method might look like:

//our user clas
public virtual ValidationError[] Validate(IRepository repository)
{
    if (repository.Exists<User>(u => u.Email == Email))
    {
        return new[] {new ValidationError("Email", "This email is already registered")};
    }
    return null;
}

Notice that I pass around an IRepository. That's just to show how you might pass infrastructure elements that your validators will need.

Back to the Client

You can now use the Validator from your WebForms or Controllers. The downloadable example shows a more proper (opinionated) MVC example, but without going into the necessary infrastructure, you might do something as simple as:

public class UserController : Controller
{
    private readonly IValidator _validator;

    public UserController(IValidator validator)
    {
        _validator = validator;            
    }

     //Everything about this action is ugly, check out the example for a better approach
    public ViewResult Register([Bind(Exclude="id")]User user)
    {
           ValidationError[] errors;
            if (!_validator.Validate(user, out errors))
            {
                ViewData["ValidationErrors"] = ToJson(errors);
                return View(user);
            }
            //todo Save the user;
            return View("RegistrationSuccessful");
    }

    private string ToJSon(ValidationError[] errors)
    {         
            var sb = new StringBuilder("init:{");
            Array.ForEach(errors, e => sb.AppendFormat("{0}:'{1}',", e.Field, e.Message.Replace("'", "\\'"));
            _validationErrors = sb.RemoveLast().Append("}").ToString();
            return _validationErrors;
    }
}

We've serialized our array of errors to JSON and stored them in our ViewData. We'll modify our jQuery plugin to handle the errros. First though, we change our jQuery call:

$('#register').validate({<%=Html.RulesFor<User>() %>, <%=ViewData["ValidationErrors"]%>});

The last change is to the initialize method, while we are hooking up each field's blur event, we'll also see if an init error exists within the options:

$fields.each(function(i, field)
{
    var $field = $(field);
    $field.blur(function()
    {
        v.validateField(this);
    });                          
    if (options.init[field.name] != null)                        
    {                            
        v.markAsInvalid($field, options.init[field.name] != '' ? options.init[field.name] : rules[field.name].tip);
    }                      
});

Conclusion

Folks, that's how easy it is to do custom validation. Granted its quite a bit of code, but there's nothing overly complex (except for maybe the reflection or jQuery plugin if you aren't familiar with either). Do stay tuned as tomorrow I'll make a sample application available for download which should help make all of this more useful.

The 7 Phases of Unit Testing

karl — Tue, 28 Apr 2009 01:02:00 GMT

1 - Refuse to unit test because "you don't have enough time"
2 - Start unit testing and immediately start blogging about unit testing and TDD and how great they are and how everyone should do it
3 - Unit test everything - make private methods internal and abuse the InternalsVisibleTo attribute. Test getters and setters or else you won't get 100% code coverage
4 - Get fed with how brittle your unit tests are and start writing integration tests without realizing it.
5 - Discover a mocking framework and make heavy use of strict semantics
6 - Mock absolutely everything that can possibly be mocked
7 - Start writing effective unit tests

Validation - Part 2 - Client-Side

karl — Tue, 28 Apr 2009 00:27:00 GMT

Validation - Part 2 - Client-Side

In Part 1 we looked at our basic framework for validation - a set of attributes, an interface, and some configuration objects. Now we'll see how these can be used in conjunction with jQuery to provide client-side validation.

As we already saw, every validation attribute has a ToJson method which serializes attributes into a dictionary. Our goal is end up with something like:

$('#saveUserForm').validate(
   {
      rules:
      {
         Email:{required:true,pattern:'.+@.+\\..+'},
         Password:{required:true,min:4,max:30}
      }      
   }
);

HtmlValidation

Before we look at the actual jQuery validate function, lets see how we can take our ValidatorConfiguration to generate this type of JSON object. Like we did with the core ValidationConfiguration, we'll cache our JSON strings per object type so that they aren't generated over and over again. Here's the core body of our class:

public static class HtmlValidation { private static readonly IDictionary<Type, string> _rules = new Dictionary<Type, string>(); public static void Initialize(IDictionary<Type, EntityValidationInfo> rules) { foreach (var rule in rules) { _rules.Add(rule.Key, BuildJson(rule.Value)); } } private static string BuildJson(EntityValidationInfo entity) { //todo } }

We initialize this our HtmlValidation class in our Application_Start:

ValidatorConfiguration.Initialize("MyAssembly", "MyAssembly.Web");  //from part 1
HtmlValidation.Initialize(ValidatorConfiguration.Rules);

Next we implement the BuildJson method:

private static string BuildJson(EntityValidationInfo entity)
{
    var sb = new StringBuilder("rules:{");
    foreach (var property in entity.Properties)
    {                
        sb.Append(property.Property.Name);
        sb.Append(":{");
        foreach (BaseValidatorAttribute attribute in property.Validators)
        {
            foreach (var kvp in attribute.ToJson())
            {
                sb.AppendFormat("{0}:{1},", kvp.Key, kvp.Value);
            }
        }
        var tip = property.Property.GetCustomAttributes(typeof(TipAttribute), true);
        if (tip != null)
        {
            sb.AppendFormat("tip:'{0}'", (((TipAttribute)tip[0]).Tip).Replace("'", "\\'"));
        }
        sb.RemoveLastIf(',');
        sb.Append("},");
    }
    return sb.RemoveLast().Append("}").ToString();
}

Nothing too fancy here. We ToJson each attribute of each property and join them into a JSON string. Those RemoveLast and RemoveLastIf methods on StringBuilder are little extension methods I use to stay sane:

public static class StringBuilderExtensions
{
    public static StringBuilder RemoveLast(this StringBuilder sb)
    {
        return sb.RemoveLast(1);
    }
    public static StringBuilder RemoveLastIf(this StringBuilder sb, char c)
    {
        if (sb.Length > 0 && sb[sb.Length - 1] == c)
        {
            sb.RemoveLast();
        }
        return sb;            
    }
    public static StringBuilder RemoveLast(this StringBuilder sb, int count)
    {
        if (sb.Length > count)
        {
            sb.Remove(sb.Length - count, count);
        }
        return sb;
    }        
}

With that done, we just need a method that makes our cached _rules accessible. I wrote this an an HtmlHelper extension for MVC, but there isn't anything MVC-specific about it. You should be able to easily get this working in WebForms as well:

public static string RulesFor<T>(this HtmlHelper html)
{            
    return _rules[typeof (T)];            
}

Now we can use that from within our aspx page/view:

$('#register').validate({<%=Html.RulesFor<User>() %>});

$.validate

Our final task in this part is to build our validate function using jQuery. I've written a 2-part primer (part 1, part 2) in the past, so I'll assume that you are familiar with jQuery. Needless to say that I'm a big fan of it and rather enjoy writing JavaScript plugins (funny how much I hated JavaScript not too long ago). We'll look at the validate plugin in chunks, and I'll provide the complete code at the end. First, a skeleton:

(function($)
{
    $.fn.validate = function(options, command)
    {
        var defaults = { };
        options = $.extend(defaults, options);
        var rules = options.rules;        
        
        return this.each(function()
        {   
            //make sure this isn't called twice
            if (this.validator) { return false; }
            var $form = $(this);    
            var $fields = $('input,select,textarea', $form);

            var v =
            {
                initialize: function()
                {              
                },
                validateField: function(field)
                {                                                              
                },            
                markAsInvalid: function($field, tip)
                {                                                     
                },
                markAsValid: function($field)
                {
                }  
            }                     
            this.validator = v;
            v.initialize();         

        });
    };
})(jQuery);

The initialize method hooks up to the form's submit event and validates each field, it also gives focus to the first invalid field:

initialize: function()
{              
    $form.submit(function()
    {
        var isValid = true;
        $fields.each(function(i, field)
        {                            
            if (!v.validateField(field) && isValid) 
            { 
                isValid = false;
                $(field).focus();
            }
        });                                                
        return isValid;
    });                    
},

By returning false when there's an error, we prevent the form from submitting. The actual validation takes place inside the validateField method. Since we only have a few simple validators, everything happens within that method:

validateField: function(field)
{                                           
    var rule = rules[field.name];                    
    if (!rule) { return true; }                    
    var $field = $(field);
    var value = $field.val();
    var isValid = true;
    if (rule.required && value.length == 0) { isValid = false ; }
    else if (rule.min && rule.min > value.length) { isValid = false; }
    else if (rule.max && rule.max < value.length) { isValid = false; }
    else if (rule.pattern && !rule.pattern.test(value)) { isValid = false; }
    if (!isValid)
    {                        
        v.markAsInvalid($field, rule.tip);                        
    }
    else
    {
        v.markAsValid($field);                        
    }
    return isValid;                    
},

The last two methods, markAsValid and markAsInvalid display or remove an actual error

markAsInvalid: function($field, tip)
{                                    
    if (tip)
    {
        var $tip = $field.siblings('.tip');
        if ($tip.length == 0)
        {                        
            $tip = $('<label>').attr('for', $field.attr('name')).addClass('tip').insertAfter($field);                        
        }
        $tip.html(tip).fadeIn();
    }
    $field.addClass('error');
},
markAsValid: function($field)
{
    $field.siblings('.tip').fadeOut();
    $field.removeClass('error');
}

if markAsValid is given a tip, it'll add and fadeIn a label next to the field. It'll also add a class named error to the field. markAsValid does the opposite - fading out the tip and removing the class.

Improvement

We'll add a slight improvement to our initialize method - not only will we hook into the form's submit event, but also each field's blur event, so that users get a more responsive feel. Luckily, all the infrastructure is already in place, so all we do is add the following code to initialize:

$fields.each(function(i, field)
{
    var $field = $(field);
    $field.blur(function()
    {
        v.validateField(this);
    });                                            
});

Code

Believe it or not, all that code make up only two classes. Since it was all broken up, here they are.

First, the HtmlValidation class (minus the StringBuilder extentions):

public static class HtmlValidationExtensions
{
    private static readonly IDictionary<Type, string> _rules = new Dictionary<Type, string>();    
    public static string RulesFor<T>(this HtmlHelper html)
    {            
        return _rules[typeof (T)];            
    }
    public static void Initialize(IDictionary<Type, EntityValidationInfo> rules)
    {            
        foreach (var rule in rules)
        {
            _rules.Add(rule.Key, BuildJson(rule.Value));
        }
    }
    private static string BuildJson(EntityValidationInfo entity)
    {
        var sb = new StringBuilder("rules:{");
        foreach (var property in entity.Properties)
        {                
            sb.Append(property.Property.Name);
            sb.Append(":{");
            foreach (BaseValidatorAttribute attribute in property.Validators)
            {
                foreach (var kvp in attribute.ToJson())
                {
                    sb.AppendFormat("{0}:{1},", kvp.Key, kvp.Value);
                }
            }
            var tip = property.Property.GetCustomAttributes(typeof(TipAttribute), true);
            if (tip != null)
            {
                sb.AppendFormat("tip:'{0}'", (((TipAttribute)tip[0]).Tip).Replace("'", "\\'"));
            }
            sb.RemoveLastIf(',');
            sb.Append("},");
        }
        return sb.RemoveLast().Append("}").ToString();
    }
}

Next, the $.validate plugin:

(function($)
{
    $.fn.validate = function(options, command)
    {
        var defaults = { };
        options = $.extend(defaults, options);
        var rules = options.rules;        
        
        return this.each(function()
        {
            if (this.validator) { return false; }
            var $form = $(this);    
            var $fields = $('input,select,textarea', $form);

            var v =
            {
                initialize: function()
                {
                    $fields.each(function(i, field)
                    {
                        var $field = $(field);
                        $field.blur(function()
                        {
                            v.validateField(this);
                        });                                      
                    });
                    $form.submit(function()
                    {
                        var isValid = true;
                        $fields.each(function(i, field)
                        {                            
                            if (!v.validateField(field) && isValid) 
                            { 
                                isValid = false;
                                $(field).focus();
                            }
                        });                                                
                        return isValid;
                    });                    
                },
                validateField: function(field)
                {                                           
                    var rule = rules[field.name];                    
                    if (!rule) { return true; }                    
                    var $field = $(field);
                    var value = $field.val();
                    var isValid = true;
                    if (rule.required && value.length == 0) { isValid = false ; }
                    else if (rule.min && rule.min > value.length) { isValid = false; }
                    else if (rule.max && rule.max < value.length) { isValid = false; }
                    else if (rule.pattern && !rule.pattern.test(value)) { isValid = false; }
                    if (!isValid)
                    {                        
                        v.markAsInvalid($field, rule.tip);                        
                    }
                    else
                    {
                        v.markAsValid($field);                        
                    }
                    return isValid;
                    
                },            
                markAsInvalid: function($field, tip)
                {                                    
                    if (tip)
                    {
                        var $tip = $field.siblings('.tip');
                        if ($tip.length == 0)
                        {                        
                            $tip = $('<label>').attr('for', $field.attr('name')).addClass('tip').insertAfter($field);                        
                        }
                        $tip.html(tip).fadeIn();
                    }
                    $field.addClass('error');
                },
                markAsValid: function($field)
                {
                    $field.siblings('.tip').fadeOut();
                    $field.removeClass('error');
                }  
            }                     
            this.validator = v;
            v.initialize();         
        });
    };
})(jQuery);

Conclusion

That's a lot of code, but hopefully it'll help you get started. In part 3 we'll look at server-side validation, and how to tie that back into the client.

Validation - Part 1 - Getting Started

karl — Sun, 26 Apr 2009 19:17:00 GMT

There are a lot of good and free frameworks to help you deal with validating user-submitted data. You can use attribute-based frameworks, such as Castle Validators or .NET 3.5 Data Annotations to decorate your objects with simple rules, and then leverage frameworks such as xVal or the MVC Validation Toolkit to enforce those attribute both on the client and server side. (For those who dislike using attributes in this manner, you could also use something like FluentValidation).

Despite these frameworks, there may be situations where you need to roll your own, which is what we're going to look at in this series. We won't be doing anything fancy, but we will look at an end-to-end custom solution.

Attributes

The first thing we need to do is decide how we'll define our rules. We'll pick a mix of attributes for simple validation cases, and an interface for more advanced/custom validation. Our goal is to end up with something like:

public interface IValidate
{
    ValidationError[] Validate(IRepository repository);
}

public class User : IValidate
{        
	private string _email;
	private string _password;

	[Required, StringLength(50), Pattern(".+@.+\\..+"), Tip("Please enter a valid email, a confirmation email will be sent.")]
	public string Email
	{
	    get { return _email; }
	    set { _email = value; }
	}
	[Required, StringLength(4, 30), Tip("Please enter your password. 4-30 characters")]
	public string Password
	{
	    get { return _password; }
	    set { _password = value; }
	}

	public virtual ValidationError[] Validate(IRepository repository)
	{
	    if (repository.Exists<User>(u => u.Email == Email))
	    {
	        return new[] {new ValidationError("Email", "This email is already registered")};
	    }
	    return null;
	}
}

We could always externalize the "Tips" for localization (or cleanliness) purposes, but this keeps everything rather simple. (If you're put off by the attributes, check outFluentValidation for an idea on how else you migth define your rules).

The foundation for our attributes is a simple base class:

[AttributeUsage(AttributeTargets.Property, AllowMultiple = false, Inherited = true)]
public abstract class BaseValidatorAttribute : Attribute
{
   public abstract IDictionary<string, string> ToJson();
   public abstract bool IsValid(object value);
}

And here's the simplest implementation:

public class RequiredAttribute : BaseValidatorAttribute
{
    public override IDictionary<string, string> ToJson()
    {
        return new Dictionary<string, string> {{"required", "true"}};
    }  

    public override bool IsValid(object value)
    {
        if (value == null)
        {
            return false;
        }
        return !(value is string) || ((string)value).Length != 0;
    }
}

The IsValid method is used to validate the object server-side. The ToJson method is used to generate a javascript object for client-side use.

Let's take a look at the StringLengthAttribute - which is slightly more complicated:

public class StringLengthAttribute : BaseValidatorAttribute
{
   private readonly int _minimumLength;
   private readonly int _maximumLength;

   public int MinimumLength
   {
      get { return _minimumLength; }
   }
   public int MaximumLength
   {
      get { return _maximumLength; }
   }

   public StringLengthAttribute(int maximumLength) : this(0, maximumLength){}
   public StringLengthAttribute(int minimumLength, int maximumLength)
   {
      _minimumLength = minimumLength;
      _maximumLength = maximumLength;
   }

   public override IDictionary<string, string> ToJson()
   {            
      return new Dictionary<string, string>
         {
            { "min", _minimumLength.ToString() },
            { "max", _maximumLength.ToString() },
         };
   }

   public override bool IsValid(object value)
   {            
      if (!(value is string))
      {
         return false;
      }            
      var length = ((string)value).Length;
      return length >= _minimumLength && length <= _maximumLength;
   }
}

There really is no limit to what you're able to do with these attributes. You can build groups of validation rules (each group might have its own tip), compare validators, or even validators that hit the DB, such as a UniqueAttribute (although we've opted to implement that via the IValidate interface).

ValidatorConfiguration

There are two problems we'll run into by using attributes - they are somewhat cumbersome to program against, and they can impact performance. Since validation is static, we can parse and cache more useful structures which we can then use when doing our actual client-side and server-side validation. We'll use two classes to represent our attributes:

public class EntityValidationInfo
{
   public IEnumerable<PropertyValidationInfo> Properties { get; private set; }
   public EntityValidationInfo(IEnumerable<PropertyValidationInfo> properties)
   {        
      Properties = properties;
   }
}
public class PropertyValidationInfo
{
   public IEnumerable<BaseValidatorAttribute> Validators { get; private set; }
   public PropertyInfo Property { get; private set;}
   public PropertyValidationInfo(PropertyInfo property, IEnumerable<BaseValidatorAttribute> validators)
   {
      Property = property;
      Validators = validators;
   }
}

The purpose behind these is that given an object type, we can lookup its EntityValidationInfo which contains a collection of properties and validators. Next we'll create a ValidatorConfiguration class with a couple static members:

public class ValidatorConfiguration
{
    private static IDictionary<Type, EntityValidationInfo> _rules;
    public static IDictionary<Type, EntityValidationInfo> Rules
    {
        get { return _rules; }
    }
    public static void Initialize(params string[] assemblyNames)
    {     
          //todo
    }
}

Our goal is to initialize ValidatorConfiguration once, on startup, and then access its Rules property as needed. The Initialize method takes in assembly names to scan for classes using validation attributes (you might have these spread out across multiple assemblies). In an ASP.NET application, we'd use the Application_Start event and use something like:

ValidatorConfiguration.Initialize("MyAssembly", "MyAssembly.Web");

Finally, the implementation of Initialize looks like:

public static void Initialize(params string[] assemblyNames)
{
   _rules = new Dictionary<Type, EntityValidationInfo>(10);
   foreach (var assemblyName in assemblyNames)
   {
       foreach (var kvp in LoadFromAssembly(Assembly.Load(assemblyName)))
       {
           _rules.Add(kvp.Key, kvp.Value);
       }                
   }
}
private static IDictionary<Type, EntityValidationInfo> LoadFromAssembly(Assembly assembly)
{
   var rules = new Dictionary<Type, EntityValidationInfo>(10);
   foreach (var type in assembly.GetExportedTypes())
   {
       var info = GetEntityValidationInfo(type);
       if (info != null) { rules.Add(type, info); }
   }
   return rules;
}
private static EntityValidationInfo GetEntityValidationInfo(Type type)
{
   var properties = new List<PropertyValidationInfo>();
   foreach (var property in type.GetProperties())
   {
       var attributes = (BaseValidatorAttribute[])property.GetCustomAttributes(typeof(BaseValidatorAttribute), true);
       if (attributes.Length == 0) { continue; }
       properties.Add(new PropertyValidationInfo(property, attributes));
   }
   return properties.Count == 0 ? null : new EntityValidationInfo(properties);
}

There isn't much magic going on here. We loop through each property, of each public type of each assembly, looking for validation attributes. If we don't find any, we simply move on to the next. If we do find them, we create new instances of our EntityValidationInfo and PropertyValidationInfo and associate that to a Type within the static _rules field.

Conclusion

We've managed to lay the foundation for actually being able to validate stuff. In Part 2 we'll look at client-side validation, and in Part 3 we'll cover server-side validation. Stay tuned.

www.asp.net costs millions to run?

karl — Sat, 25 Apr 2009 15:02:00 GMT

For those who missed it, Joe Stagner commented that www.asp.net costs millions of dollars to run. Since Joe was a little vague about exactly what that meant, I'm going to assume that he's talking about straight-up yearly hosting costs (because it was in the context of hosting, and if that was a monthly cost it would be 12-times more incredible than the already incredible yearly cost). I find this either highly doubtful, or really worrisome.

So let's look at what $83 000 a month of hosting gets you. Now, there are tens of thousands of possible hosting companies. My top picks would be Softlayer, Peer1, DataPipe or LiquiqWeb - all of which have the right blend of price and reliability (Softlayer and DataPipe are always on Netcraft's list of most reliable hosting providers). We're talking real quality hosting here. For about $350/month, I can get (more or less) the following server from any of these places:

Quad Core Xeon 5335 Clovertown @ 2.00GHz
2GB of ram
2x250GB SATA RAID 1
2TB bandwidth

You'd likely want different servers for different roles - such as more RAM and SCSI drives on your database servers, but for simplicity sake, lets keep everything homogenized.

That $83 000 buys you roughly 240 such servers - fully managed. Am I the only one who thinks that's a lot? Consider too that this is retail price, surely Microsoft, with such a large order, could get drastically lower prices (and the fact that they heavily promote their existing hosting company). Now, it's possible that Joe was talking about more than just www.asp.net, and possibly all of the "Microsoft Community" sites (channel 9, IIS 7.0, silverlight.net...). That might make things slightly more reasonable. However, I remain both skeptical, and very curious about the details of this server farm. How many memcached servers are in play? Is database sharding used? Are lightweight HTTPd servers or CDN networks used for static content?

What worries me is that if it costs Microsoft millions of dollars to run its own community sites, with its bulk-purchasing power and free Microsoft licences what chance to the rest of us have? It suddenly becomes obvious why IIS has a lower market penetration amongst popular sites (19%) than its average market penetration (29%). Its also obvious why that market penetration has gone down over the past couple years.

Finally, I'm curious if they do any carbon offsetting. I doubt this setup is even close to achieving google's amazing PUE of 1.13.

Is ASP.NET MVC a half-baked solution?

karl — Fri, 24 Apr 2009 17:45:00 GMT

Earlier this morning, I commented on twitter that I was starting my 2nd full-fledged ASP.NET MVC project, and was finding myself writing a massive amount of infrastructure code. Ironically, having to write so much infrastructure code was a major complaint I had against both Oxite and Kobe. A number of other twitter-folk replied with similar thoughts. There isn't a day that passes that I don't think to myself using ASP.NET MVC is a mistake.

Let's be clear, ASP.NET MVC is an improvement over WebForms - and those developers refusing to leverage ASP.NET MVC (or something else) for new projects are simply lazy. WebForms is full of leaky abstraction, really tight coupling, and ridiculous concepts - such as viewstates and postbacks.

Being able to write complex systems more cleanly is a good start, but given where web development stands in general, and other platforms specifically, ASP.NET MVC lags far behind (Perl being the only one I can think of which is worse).

There's little question that a big part of the problem is that this is really a VC stack - there is no thought, no support, and no tools for the Model. When you compare the thousands of lines you'll end up writing for your repository/dal/linq/nhiberate to other MVC stacks (which commonly only require that your models to inherit from 1 class), you're already at a serious productivity disadvantage. But the true impact is actually much worse - you lose any cohesiveness of purpose through the controller and views. There is no way to generate HTML labels from model properties, or client side validation. In frameworks such as Akelos, RoR or Django everything flows outwards from the model which allows those frameworks to provide cohesive and integrated tools, not just HtmlHelpers, ActionResults and Routing (which is all you get from ASP.NET MVC).

You end up having to do a whole lot of plumbing on both sides of your system. Alternatively, you can leverage some OSS tools, like MvcContrib, FluentNHibernate, xVal, Castle.*, jQuery. While all of those tools are really great, they too lack any cohesiveness. For example, trying to modify the jquery.validate hook from xVal which relied on modified Castle.Validators (to make them work on FluentHTML) was simply too much work for me. I considered using FluentValidation, but the only reference to xVal integration was a blog post.

I'm a huge fan of these tools, but ASP.NET MVC + 7 different OSS projects simply can't come close to competing with the elegance, ease of use, and completeness of a well-managed and controlled project like Web2Py (for one of the many examples).

There is some good news, and that's that a lot of this "infrastructure" is reusable, which makes projects like S#arp Architecture possible. However, I'm still sceptical that these projects can truly succeed against better integrated frameworks.

There will be those that defend ASP.NET MVC by saying that it's still very young. Microsoft only has itself to blame for being so slow and hard-headed for so long. Others will be thankful that ASP.NET MVC is less opinionated, and thus more flexible, than most other MVC stacks. I agree with this, but only because they ought to have an System.Mvc.Core assembly which contains the open-and-flexible frameworks (yet still provide much more than they currently do), as well as a System.Mvc assembly which is much more opinionated and thus provides even greater functionality.

When you look at the popular web frameworks for web 2.0 style sites, ASP.NET MVC and WebForms are backed by the biggest and richest company (by several orders of magnitude) yet remains the least popular. ASP.NET MVC offers less features, requires more knoweldge, and has lower productivity, despite Microsoft having a cafeteria staff that's probably 10x the size of PHP, Rails and Django's entire development teams combined. (Even more ironic, www.asp.net is the only major web-framework website to have ads on it...none of the open source websites do...how screwed up is that?)

Tidbit - svn:externals

karl — Mon, 20 Apr 2009 13:24:00 GMT

Not too long ago, I had a need to tie a number of svn repositories together. What I wanted was to create a 1 repository with a number of sharable assets (images, language files, 3rd party libraries, shared code) and have other repositories somehow link to it.

The way I had handled this in the past was simple: checkout each svn repository and cross reference them. As an example, my directory structure would look something like:

AssetRepository/references/xunit.dll
Project1Respository/Project1.sln
Project1Respository/Project1.Tests/Project1.Tests.csproj

I would then add a reference to the xunit.dll from Project1.Tests, which would use the following relative path: ../../AssetRepository/references/xunt.dll.

This solution works great, except it imposes a structure that each team-member must adhere to. In this case, all repositories must be at the same level within the same folder.

Luckily, Subversion has a better solution, the svn:externals property. svn:externals essentially creates a shortcut/link from a folder in one repository to some other repository (or to another place within the same repository). Once 1 person sets it up, it just works for everyone else – they'll never know that AssetRepository even exists.

How do you do it? If you are using TurtoiseSVN, right click on the Project1Repository and pick “Properties” from the TurtoiseSVN context-menu. Add a new entry with the name of "svn:externals" and a value to the local path + SPACE + external repository. In our case, value would look like:

References svn://AssetRepository/references

That's all there is to it. You checkout the Project1Repository, you'll end up with an extra folder:

Project1Respository/Project1.sln
Project1Respository/Project1.Tests/Project1.Tests.csproj
Project1Respository/References/xunit.dll

You might think that you need to create an empty 'References' folder at the root of your Project1Repository – don't. The local path specified in your svn:externals definition should not physically exists within that repository.

That's all there is to it. Other developers can now checkout/commit without ever knowing how things are set. If they change a file within the References/ folder, it'll commit to the proper repository, namely AssetRepository.

You can learn more from the red-book