With the many posts that I’ve done on the Reactive Extensions for both JavaScript and .NET, I’ve covered a wide variety of the basics as well as some of the deeper stuff.  This time, I’d like to get back to the basics that I do when I give a talk on Rx in person, and this time explaining some of the problems we face today with reactive programming in general, asynchronous and event-based.

Why Do We Need It?

Let’s go into an example of using the event-based asynchronous programming methods that became a bit more prevalent when .NET 2.0 came around, especially with the introduction of WCF in the .NET 3.0 timeframe.  This was an alternative to the other methods of asynchronous programming which included the Begin/End Pattern and using WaitHandles.  One distinct advantage it did have was for the ability to report progress, report incremental results or state changes that the others could not provide.  But with it, comes many downsides as well which we’ll go into and where the Reactive Extensions can help. 

Typically, we’d have some form of asynchronous method which has no return value and takes in any number of parameters, and a given user token which would allow us to correlate our token with what comes in the user state to make sure we’re looking at the right result. 

public void DoSomethingAsync(object token) 
{
    // Kick off something async
}

And we’d also have some form of EventArgs which would encompass our result and exception should one occur and optionally a cancelled flag should we need to do so and be notified that it happened.  Then we’d also have our EventHandler which is then called when our asynchronous method is complete.

public class DoSomethingCompletedEventArgs : EventArgs 
{
    public string Result { get; set; }
    public bool Cancelled { get; set; }
    public Exception Error { get; set; }
}

public event EventHandler<DoSomethingCompletedEventArgs>
    DoSomethingCompleted;

To give you an idea of some of the problems we face with this pattern, let’s go over an example using the System.Net.WebClient class and downloading a string asynchronously via the DownloadStringAsync method.  Let’s enumerate some of the challenges in the code below.

var uri = new Uri("http://search.twitter.com/search.json?q=4sq.com");
var client = new WebClient();

client.DownloadStringCompleted += (sender, args)
{
    // TODO: Get the only one we want?
    // TODO: Check for cancel?
    // TODO: Check for exception?
    // TODO: Check for result?
    
    // TODO: What about more composition?
};

client.DownloadStringAsync(uri);

// TODO: How to unsubscribe
client.DownloadStringCompleted -= ???

The first challenge is how to make sure we get the value we want.  After all, if multiple calls are made to this WebClient instance, we’ll need to correlate the user state with our token.  Next, we need to check for cancellation and do something when that happens.  After checking for cancellation, we might also want to check for exceptions and handle them appropriately as well.  And finally we get our result, now what do we do with it?  Better yet, now that I have the result, how can I compose it with another asynchronous call?  Finally, when using lambdas for subscribing to events, I can’t unsubscribe our inline lambda, so that could cause an issue as well.  Making some of those changes to handle our cases properly, we end up with code that looks like the following:

var uri = new Uri("http://search.twitter.com/search.json?q=4sq.com");
var client = new WebClient();
var token = new object();

DownloadStringCompletedEventHandler h = (sender, args)
{
    // Check if it's ours
    if (token != args.UserState) return;

    if (args.Cancelled)
    {
        // Do something to say we stopped
    }
    else if (args.Error != null)
    {
        // Do something with that error
    }
    else
    {
        // Well, now what?
    }
};

client.DownloadStringCompleted += h;
client.DownloadStringAsync(uri, token);

// When we're ready to clean up
client.DownloadStringCompleted -= h;

We solved a good majority of the problems with a lot of boilerplate.  But within all this boilerplate, we’re losing sight of our core goal here, which is to get our result and do something with it, especially when that next call happens to be to another asynchronous call.  For example, what if we want to download a string from somewhere, transform it into JSON, modify the results and then upload somewhere else?  Our code might look like the following should we use the Reactive Extensions for .NET to solve it.

var client = new WebClient();
var searchUri = new Uri("http://search.twitter.com/search.json?q=4sq.com");
var uploadUri = new Uri("http://localhost:8080/uploads");

IObservable<Unit> query =
    from result in client.DownloadStringAsObservable(searchUri, new object())
    let transformed = TransformResult(result)
    from upload in client.UploadStringAsObservable(
        uploadUri,
        "POST",
        transformed,
        new object())
    select upload;
    
var subscription = query.Subscribe(
    x => {}, // Nothing to do
    exn => 
    {
        // Do something with the exception
    }, 
    () => 
    {
        // Indicate we're finished
    });

What we’re able to do is take operations that would have required quite a bit of code in terms of boilerplate, and allow for composition through LINQ via the Reactive Extensions for .NET.  But, how do we get there? 

How Do We Get There?

What we need to do at this point is to transform our asynchronous call from above and turn it into an IObservable<T> instance.  Using the FromEvent which we’ve covered earlier, won’t cut it because ultimately, we want to combine the asynchronous call with the event handler which is invoked upon completion, which FromEvent cannot provider.  Instead, we want to focus our effort into the Observable.Create and Observable.CreateWithDisposable methods to help us.  This allows us to create IObservable<T> instances from the subscribe implementation.  We can either choose the CreateWithDisposable method which gives us an IObserver<T> instance and we return a concrete IDisposable instance, or we call Create in which are given the same IObserver<T> instance, but we return an Action instead of the IDisposable instance.  Let’s look at those signatures below:

public static IObservable<TSource> CreateWithDisposable<TSource>(
    Func<IObserver<TSource>, IDisposable> subscribe
)

public static IObservable<TSource> Create<TSource>(
    Func<IObserver<TSource>, Action> subscribe
)

Using the Observable.Create method, let’s create a method called DownloadStringAsObservable extension method to handle the asynchronous request.  Inside the Observable.Create, we’ll create our DownloadStringCompletedEventHandler where we can check for our user token, handle cancellation by calling observer.OnCompleted, handle exceptions via observer.OnError and finally handle our result by calling observer.OnNext and observer.OnCompleted to indicate we are finished.  We then take this handler and listen to the DownloadStringCompleted event.  We want to be able to catch any errors in our DownloadStringAsync should it throw an exception, so we put that in a try/catch block and send the Exception to observer.OnError.  Finally, Being the good citizens we are, we want to clean up after ourselves by removing our handler from the DownloadStringCompleted event when we dispose of our subscription.

public static IObservable<string> DownloadStringAsObservable(
    this WebClient client, 
    Uri address,
    object userToken)
{
    return Observable.Create<string>(observer =>
    {
        DownloadStringCompletedEventHandler handler = (sender, args) =>
        {
            if (args.UserState != userToken) return;
    
            if (args.Cancelled)
                observer.OnCompleted();
            else if(args.Error != null)
                observer.OnError(args.Error);
            else
            {
                observer.OnNext(args.Result);
                observer.OnCompleted();
            }
        };
    
        client.DownloadStringCompleted += handler;
        
        try
        {
            client.DownloadStringAsync(address, token);
        }
        catch (Exception ex)
        {
            observer.OnError(ex);
        }
        
        return () => client.DownloadStringCompleted -= handler;
    });
}

The sample logic applies to uploading a string as well by wrapping the UploadStringAsync method as we do below.

public static IObservable<Unit> UploadStringAsObservable(
    this WebClient client, 
    Uri address, 
    string method, 
    string data,
    object userToken)
{
    return Observable.Create<Unit>(observer =>
    {
        UploadStringCompletedEventHandler handler = (sender, args) =>
        {
            if (args.UserState != userToken) return;
    
            if (args.Cancelled)
                observer.OnCompleted();
            else if (args.Error != null)
                observer.OnError(args.Error);
            else
            {
                observer.OnNext(new Unit());
                observer.OnCompleted();
            }
        };
    
        client.UploadStringCompleted += handler;
        
        try
        {
            client.UploadStringAsync(address, data, method, token);
        }
        catch(Exception ex)
        {
            observer.OnError(ex);
        }
        
        return () => client.UploadStringCompleted -= handler;
    });          
}

By using these tokens as we do above, we could easily wrap the DownloadProgressChanged event as well filtering for our token as well.

public static IObservable<DownloadProgressChangedEventArgs> 
    DownloadProgressChangedAsObservable(
        this WebClient client,
        object userToken)
{
    return Observable.FromEvent<DownloadProgressChangedEventArgs>(
            client,
            "DownloadProgressChanged")
        .Where(x => x.EventArgs.UserState == userToken)
        .Select(x => x.EventArgs)
}

What we have done here is taken a less than optimal event-based asynchronous programming model and moved it to use Reactive Extensions for .NET so that we can take care of uniform cancellation checking, error handling and allow for composition.

Conclusion

Asynchronous programming is something that’s creeping up more and more to .NET developers, especially with Silverlight, Windows Phone and so forth.  Unfortunately, dealing with asynchronous programming has been and continues to be hard.  By using the Reactive Extensions for .NET, we have ways to build bridges from the old asynchronous programming models, to a more composable form where we can uniformly handle exceptions, cancellation and so forth.

So with that, download it, and give the team feedback!

Under 500 lines of code before the client

http://github.com/gregoryyoung/m-r

I will write a blog post about some of the stuff in it but its pretty straight forward.

Smartphones and all other kinds of mobile devices are everywhere, almost everybody has one. Not just for entertainment but also as a client for all kinds of enterprise software. Making choices when developing desktop software was easy, the main discussion usually was restricted to deciding which Windows version to target. Making choices when developing software for mobile is harder. No platform is dominant and each platform has other tools and languages. The nice thing with the Windows Mobile platform is that you can use the same tools and languages (read C# and the .net platform) as used for desktop applications. The nice thing about Apples iPhone is that it has an incredible UI, is also widely used and has a good possibility for developing commercial applications.

Wouldn’t it be nice to have the best of both worlds ? The quality of the tools and language of Windows Mobile and the pleasure of use of an iPhone  ?

A small Windows Mobile rant

I have to get something of my chest first. This is an important reason, besides the iPhone just being sexier :), to  look further than WM.

Building applications for Windows Mobile is, thanks to the .net compact framework, no big deal. For a couple of years my Windows Smartphones were a nice and reliable target. But instead of getting better the Windows Mobile platform has become worse in the everyday use, despite all the “integration” in Windows. Let me sum up my complaints over the last year:

• Since Vista many a smartphone is no longer recognized when connected over USB. Switch to bluetooth ?
• After a bluetooth connection with my laptop I have to reboot the phone before it will connect to anything else (like my old a plain vanilla carkit)
• Office 64 bit does not sync with any Windows Mobile device at all Switch back to Office 32 bit ?

All of these complaints are well known and acknowledged. And will not be fixed. Period. (None of them are a problem when trying to sync an iPhone with MS Outlook) So you are isolated more and more with Windows Mobile. Yes things will be better with Mobile 7. But that’s somewhere in the future and there is no way to migrate existing hard- and software.

The good thing with WM7 is that all software development is based on Silverlight. Which has proven to be a nice development environment with a very good subset of the .net framework which encapsulates the platform. You only have take care about the form factor of the hardware and can forget all about the dreaded OS. For now it’s time to look at the iPhone.

Wrapping up iOS

At first sight the tools for creating iOS (the OS for the iPhone and iPad) applications don’t look very appealing. The API is rich but the language (objective C) is not very friendly. Worst of all you have to do your own memory management. Memory is freed using ref-counting, just like in COM. I have had my share of that working with Delphi in COM and avoid it like the plague ever since. The main problem with refcounted objects is what happens with them when they are passed as a parameter to some other piece of code. Without a clue what will happen to their count while spending some time there. Memory leaks as well as null pointers are usually the result. Trying to avoid that leads to ugly changes in the software’s design, only trying to satisfy memory management.

Here the MonoTouch framework comes to the rescue. It wraps up the iOS framework in a .net style library and adds a C# compiler for iOS. All external resources are wrapped up in the IDisposable pattern. Over the years a garbage collector and IDisposable have proven to be, despite the initial skepticism, quite reliable. And have lead to code which is far simpler.

Building an iPhone app in C#

You need several things to do this, the first is a Mac with the newest version of OS X. It does not have to a monster, for me the smallest Macbook is by far powerful enough. Monotouch runs as a part of MonoDevelop, which looks like a simple Visual Studio for the Mac.

Simple, but all the essentials are included. Monodevelop has integrated nUnit support and refactoring. Don’t expect too much of the latter, it pales compared to that of VS. Not to mention Resharper. Typing code in MonoDevelop is a new form of sports on itself. The difference in the navigation keys itself is enough to drive you nuts in the beginning. And, without Resharper, there is so much more which you have to type.

The good part is that running and debugging the app works like a charm. The iPhone emulator works well, including network support. Which can be another PITA developing for Windows mobile. Just for fun, try watching a Youtube movie in the emulator. The debugger is a way of exploring what’s inside an iPhone. The stability of the environment is good, it takes inspecting null pointers to non managed resources to hang up MonoDevelop. With “Force Quit” in the apple OS (comaparable to the Windows Task Manager) a quick restart of the tool is easy.

Apple support

The App Store is a way to sell your apps. Before an app is accepted for resale it is tested and scanned by Apple. A lot has been said about Apple banning apps not created with Apple’s own SDK. Monotouch apps compile to native iOS code and Apps built with Monotouch are reported to be accepted to the appstore. Working for the appstore takes some investments. Beside the mac hardware and an entrance fee to the store there is $500 for a Monotouch licence to deploy your apps to an iPhone or the store. Apps built with the free (trial) version only deploy to the emulator.

Taking the bait

Having explored Monotouch I have taken the bait. As with every new thing the first steps on the curve are hard. There are a lot of resources on the web, the quality varies. As a first step I started with refactoring the examples. By hand… Soon the result was astonishing. Most examples are in a very chatty syntax; refactoring the noise to auto-properties soon revealed the intention of most of the code. And quite soon I found my way. I’ve got the iPhone in my .net. For now I can only advise you to, in case you have a Mac, take a look at Monotouch. Enjoy! More on my experiences later.

What if software development was taught like we teach a language? (This question led me to following thoughts, I suck at grammar and am not saying this is how we actually learn, these are just the thoughts that came to me while pondering that question.) That we are first taught the most fundamental of basics in a language, the ABCs, the sounds of our native tongue. Through this most primitive of understanding we start to grow our proficiency.

I didn't attend university courses on software development but I would guess that it doesn't start with learning about AND/OR/NOT constructs. Not that we need to spend a crap ton of time on this topic but do we spend any? Is there any value in it, when I read the Pattern on the Stone I found understanding the basics to be fascinating. It's understanding things at that low level that I feel has allowed me understand things at the higher levels.

As we progress in our understanding, we should move away from using primitive language structures, and start to develop a level of comprehension that goes beyond simple sentences, and one word responses. We need to learn how to compose prepositional sentences, and transitions. When I read Object Thinking I started to see the world as the designers of object oriented languages wanted me too. Now I just need to find the books on functional and other language styles.

Learning how to compose the chapters and novels of our software development can prove to be an even larger challenge for us in the software development community. It takes an awful long time to build up a program that has enough complexity to warrant being compared to a 300 or 400 page book. We study the architecture of our language from the time we are infants, how can we embed the concept of architecture in programming through out the whole of the education? When our projects are relatively small by comparison the large programs we will become responsible for when we move out to the real world?

Though I have studied many other languages than English (German, Spanish, Japanese, and Chinese) I am not sure its helped me really understand English. But my friends who actually are proficient in there non-native languages say that it helps. Are we asked to study multiple languages? This is an area I am sure is better addressed, but are we studying the differences in the languages. The values enshrined in .Net over Perl over Python over Ruby over Assembly etc? To see why some languages suck at some tasks and other languages are great for other tasks. Are we taught how to build a simple language, and then a simple language on top of another language, and then a DSL. To see the various amounts of effort required to achieve the simplest of things?

Where are the art and craft classes of software development, where the value is on the aesthetic of the program and on the code. Where is the poetry class of software design where we focus on writing that one-liner of pearl? Where is the class that teaches us the code smells and how to fix them.

What about the journalism class where we learn to communicate to the masses. Where we learn to speak to an audience larger than one passionate reader. To publish on a deadline, to not write above our readership. To realize that even though we may have an extended vocabulary and that is some thing that we should pride our selves on, that we must remember that people other than us are going to be reading it and that the goal is comprehension not on flexing our vocabulary might.

Ok, well I think I have pushed this metaphor about as far as I want to. Thank you for reading, I hope that it inspires you to think about what you will learn next.

-d

One popular aspect of the tool NDepend is the integration of the tool into the Continuous Integration build process. Every morning the team leader gets an up-to-date report telling if some coding rules written with CQL have been violated within the last 24h.

With NDepend v3 integrated in Visual Studio 2010, 2008 and 2005, we wanted to provide a smoother answer to the critical scenario of when a developer violates a rule. We think that waiting till the day after to discover that a rules get violated is too long, it is not enough agile-oriented. We wanted CQL rules violation being obviously warned as soon as possible, right within Visual Studio.

The idea of what we’ve done with NDepend v3 is that all CQL rules get checked into Visual Studio as soon as one or several VS projects have been (re)compiled.

The developer gets permanently informed of CQL rules validation status thanks to a progress circle we added in the bottom-right corner of the Visual Studio window.

•   green, means that all CQL rules are validated,
•   yellow, means some activated CQL rules are violated, 
•   red, means some activated CQL rules don’t compile and (eventually) some activated CQL rules are violated, 
•   blue + progressing, means that the code is currently analyzed by NDepend,
•   grey, means that no NDepend project is currently loaded in VisualStudio. 

Hovering the circle with the mouse shows a form summarizing all CQL rules status + a pointer to Show CQL Explorer panel:


The CQL Explorer lists all rules grouped by categories…



…and by clicking a rule you can jump to the CQL edition Windows, where you can edit the rule and see culprit methods or classes that violate the rule.


Of course double clicking a culprit method or class lets jump to its source code declaration. And if the application analyzed by NDepend spawns several VS solutions, and these solutions are currently opened in several VS instances, the code element gets edited within the adequate VS instance, the one that contains the VS solution that contains the code element. This behavior at first glance might surprise and even look like a bug. But with the habits it becomes a powerful time-saver trick. And this multi-VS-instances communication trick is made possible thanks to the Application-Wide code analysis of NDepend.

CQL rules validation phase is fast. The performance challenge was to make this happens almost instantly to avoid slowing the developer machine. Hopefully for a large 100K Lines of Code application, code gets re-analyzed and 200 CQL rules can get checked, all within 3 seconds after the (re)compilation of one or several .NET assemblies. These fast performances were made possible thanks to the development of a new technology of incremental code analysis. With incremental code analysis, only modified code gets re-analyzed. I can attest that this was extremely challenging and complex development!

Finally, in the form shown on progress circle hovering with mouse, there are 2 links Analysis Execution and Analysis Refresh in VS.

• The option panel Analysis Refresh in VS let’s customize when an NDepend analysis should occurs.

• The option panel Analysis Execution let’s finally customize the analysis process (incremental/full analysis, in/out process, report generation…).

Something that we are thinking of implementing for the future would be an option to forbid check-in if not all CQL rules are green. The development complexity comes from the multitude of Source File Management technologies (including TFS) to support. Any comment on this potential feature is welcomed dear reader.

We’ve come a long way in the series on the Reactive Extensions for JavaScript.  After spending some time with Ruby and with extension points in other libraries, let’s step back to some of the basic operators again.  This time, let’s talk about how we can buffer our input, which is to say we can put our observable values into a buffer based upon either time or by count.  This means that instead of flooding our system with calls to OnNext, we instead fill up a buffer based upon the given criteria of time or count, and then call OnNext with that value.  Let’s look deeper into how we can use this in today’s post.

Before we get started, let’s get caught up to where we are today:

• Introduction
• Creating Observables
• Creating Observers
• jQuery Integration
• Composing Callbacks
• From Blocking to Async
• Wikipedia Lookup
• Composing Deeper
• Bing Maps and Twitter Mashup
• Drag and Drop
• jQuery Live Event integration
• jQuery AJAX integration
• A Separation of Concerns
• Aggregates – Part 1 and Part 2
• Join Operators
• Going “Parallel” with ForkJoin
• Refactoring a Game
• Async Method Chaining
• Custom Schedulers
• Countdown Timers
• Wrapping the Dojo Toolkit
• MooTools Integration
• Conditional Operators
• Looping Operators
• Error Handling Part 1 and Part 2
• Rx and Dojo Deferred

Buffering Output

In a traditional pull-based model of synchronous programming, you could have scenarios where yielding the next value could be a potentially expensive blocking operations.  Contrasting it to the reactive push based model, we could get flooded with requests as our systems are eagerly sending us data as fast as it can.  Potentially this can be a problem which could overload our system.  To compensate for this, we have with the Reactive Extensions (for both .NET and JavaScript), a notion of buffering, which is to say based upon some criteria, we can buffer the results and return them as an array.  We have the choice of either buffering by count threshold, or by a given time span.  Once the criteria has been exceeded (or we reach the end), we have the values yielded to us at once in array form.  Let’s first look at the buffering by count.

…With Count?

The first buffering mechanism we have with the Reactive Extensions is by count.  To make use of this functionality, we can call the BufferWithCount method which takes a count, and optionally a number to indicate how many you want values to skip in between buffers being yielded.  This method then returns to us an array of values inside an observable sequence.

// count: the size of the buffer
// skip (Optional): how many values to skip
// returns: Observable<Array<T>>

Rx.Observable.prototype.BufferWithCount =  function(
    count, 
    skip);

Let’s look at this in action.  We’ll take an array with ten numbers and buffer them every two values, so in all we should have yielded to us five arrays.  We’ll iterate through our buffer and indicate its value and position in the buffer.

Rx.Observable.FromArray([1, 2, 3, 4, 5, 6, 7, 8, 9, 10])
    .BufferWithCount(2)
    .Subscribe(
        function (next) {
            $.each(next, function (index, value) {
                $("<p/>").html("Buffered: " + index + "-" + value)
                    .appendTo(results);
            });
        });

Below is the result of the BufferWithCount specifying only the count and not the skip count.  I have highlighted how many arrays you will get.

Buffered: 0-1
Buffered: 1-2 // one
Buffered: 0-3
Buffered: 1-4 // two
Buffered: 0-5
Buffered: 1-6 // three
Buffered: 0-7
Buffered: 1-8 // four
Buffered: 0-9
Buffered: 1-10 // five

Next, we’ll try specifying the skip count, which is to say how many elements we should skip.  For example, if I have a buffer set at two and then I specify four as the skip value, then it will yield 1 and 2 and then skip 3 and 4 before then yielding to us 5 and 6.  Let’s show how that code might look:

Rx.Observable.FromArray([1, 2, 3, 4, 5, 6, 7, 8, 9, 10])
    .BufferWithCount(2, 4)
    .Subscribe(
        function (next) {
            $.each(next, function (index, value) {
                $("<p/>").html("Buffered: " + index + "-" + value)
                    .appendTo(results);
            });
        });

And now we can sure enough see our results where we yield only three arrays this time due to us skipping 3, 4, 7 and 8.

Buffered: 0-1
Buffered: 1-2 // one
Buffered: 0-5
Buffered: 1-6 // two
Buffered: 0-9
Buffered: 1-10 // three

But this of course isn’t the only mechanism we have in our arsenal.  Let’s now talk about buffering with time.

…With Time?

In addition to buffering with count, we have a notion of buffering with time, which we can achieve with the BufferWithTime method.  This method takes in a timespan in milliseconds, an optional time shift, which acts like the skip from BufferWithCount, and an optional Rx Scheduler.  This then returns to us, as before, an Observable of an array.

// timeSpan: time span in milliseconds to buffer
// timeShift (Optional): time to skip listening in milliseconds
// scheduler (Optional): a custom scheduler
// returns: Observable<Array<T>>

Rx.Observable.prototype.BufferWithTime = function(
    timeSpan, 
    timeShift, 
    scheduler);

To show this in action, we’ll hop back to some of our knowledge gained in our custom schedulers post.  We’ll create our custom DelayedScheduler based upon a given delay time.  This will help take an existing array and then space it out over time.

var delayedScheduler = Rx.DelayedScheduler = function (delay) {

    return new Rx.Scheduler(
        function (action) {
            var id = window.setTimeout(action, delay);
            return Rx.Disposable.Create(function () {
                window.clearTimeout(id);
            });
        },
        function (action, dueTime) {
            var id = window.setTimeout(action, dueTime);
            return Rx.Disposable.Create(function () {
                window.clearTimeout(id);
            });
        },
        function () {
            return new Date().getTime();
        });
};

With the scheduler now in place, we can take our array and delay each item by 100 milliseconds by calling the FromArray with our scheduler.  We can then buffer it with a given timeout of 300 milliseconds and then subscribe much as we have before.

Rx.Observable.FromArray(
        [1, 2, 3, 4, 5, 6, 7, 8, 9, 10], 
        new Rx.DelayedScheduler(100))
    .BufferWithTime(300)
    .Subscribe(
        function (next) {
            $.each(next, function (index, value) {
                $("<p/>").html("Buffered: " + index + "-" + value)
                    .appendTo(results);
            });
        });

This will yield to us some interesting results in that we have four arrays yielded to us, some with three elements and some with only two due to the timing.

Buffered: 0-1
Buffered: 1-2 // one
Buffered: 0-3
Buffered: 1-4 
Buffered: 2-5 // two
Buffered: 0-6
Buffered: 1-7
Buffered: 2-8 // three
Buffered: 0-9
Buffered: 1-10 // four

Now, let’s try with a time shift specified.  In this example, we’ll set our timeout once again as 300 milliseconds and our shift of 100 milliseconds, which should then allow us to skip the numbers 3 and 7 based upon our timing.

Rx.Observable.FromArray(
        [1, 2, 3, 4, 5, 6, 7, 8, 9, 10], 
        new Rx.DelayedScheduler(100))
    .BufferWithTime(300, 400)
    .Subscribe(
        function (next) {
            $.each(next, function (index, value) {
                $("<p/>").html("Buffered: " + index + "-" + value)
                    .appendTo(results);
            });
        });

And sure enough that turns out to be the case as we have three arrays yielded to us, missing both the 3 and 7.

Buffered: 0-1
Buffered: 1-2 // one
Buffered: 0-4
Buffered: 1-5
Buffered: 2-6 // two
Buffered: 0-8
Buffered: 1-9
Buffered: 2-10 // three

Now we shouldn’t be limited to an either or at this point, why not have the best of both worlds?

…With Both?

As we’ve already covered both buffering with count and time, let’s cover how we can get the best of both worlds and when either the timeout or the buffer size has been hit, then yield us the values in the buffer.  We can do that through the use of the BufferWithTimeOurCount which we specify the time in milliseconds, a count, and an optional Rx Scheduler.  This then yields to us our Observable of an array.

// timeSpan: time used to determine buffer size
// count: count used to determine buffer size
// scheduler (Optional): optional scheduler
// returns: Observable<Array<T>>

Rx.Observable.prototype.BufferWithTimeOrCount = function(
    timeSpan, 
    count, 
    scheduler);

To show this in action, I’m going to need another custom scheduler, but this time completely random as to allow me to show off both behaviors.  We’ll take much of the above code from our DelayedScheduler and instead create a random number up to the delay parameter.  Then, every time a new value is to be yielded, then we delay that value by the given random number.

var randomScheduler = Rx.RandomScheduler = function (delay) {

    return new Rx.Scheduler(
    function (action) {
        var randomnumber = Math.random() * (delay + 1) << 0;
        var id = window.setTimeout(action, randomnumber);
        return Rx.Disposable.Create(function () {
            window.clearTimeout(id);
        });
    },
    function (action, dueTime) {
        var id = window.setTimeout(action, dueTime);
        return Rx.Disposable.Create(function () {
            window.clearTimeout(id);
        });
    },
    function () {
        return new Date().getTime();
    });
};

Let’s show this off by taking our standard array that we’ve used so far and use our customer RandomScheduler of a maximum of 100 milliseconds.  We’ll then buffer it by either a timeout of 200 milliseconds or 4 items and then subscribe like normally.

Rx.Observable.FromArray(
        [1, 2, 3, 4, 5, 6, 7, 8, 9, 10],
        new Rx.RandomScheduler(100))
    .BufferWithTimeOrCount(200, 4)
    .Subscribe(
        function (next) {
            $.each(next, function (index, value) {
                $("<p/>").html("Buffered: " + index + "-" + value)
                    .appendTo(results);
            });
        });

We’ll see in our results that we’ll hit the timeout quite a few more times than we hit with the count, but it shows that we do indeed hit both the timeout and count buffers.

Buffered: 0-1
Buffered: 1-2 // timeout
Buffered: 0-3
Buffered: 1-4 // timeout
Buffered: 2-5
Buffered: 3-6 // count
Buffered: 0-7
Buffered: 1-8 // timeout
Buffered: 0-9
Buffered: 1-10 // timeout

And there you have it, a simple walkthrough of using buffers with the Reactive Extensions for JavaScript (and if you squint real hard, .NET as well).

Conclusion

Dealing with asynchronous programming has been in the forefront of many minds in the JavaScript community.  At JSConf, there were several examples of frameworks trying to get around the idea of callbacks and instead lean more towards composition.  By utilizing the Reactive Extensions for JavaScript, we’re able to compose together asynchronous and event-based operations together and transform the results in interesting ways.

In the reactive programming world, we could easily have situations where we have systems pushing values at us faster than we would like.  To compensate for this, we have a notion of buffering in the Reactive Extensions which allow for us to specify either count buffers, timeout buffers, or heck, even both.  This gives us some much needed flexibility on how we process the data that is coming at us.

So with that, download it, and give the team feedback!

Tonite I came across this really nice post from Kelly (@kellabyte) on how passion drives her to continually learn and explore. She also talks about how a simple idea ends up igniting the fires and quickly spawns into putting in a lot of late nights :-) Kelly also talks about the challenge of finding an environment that lets you fulfill those desires as part of your day job rather than just in the off hours. I really appreciate Kelly’s honesty and passion, it shines through in her article.

As a side note, I can totally resonate with what Kelly’s saying. I was there for over 10 years. At each place I was trying to find my “groove”, but none of the places felt quite right. Flash forward and I can say that persistence, resilience and relentlessness finally did pay off (fortunately). Certainly there were many tradeoffs in getting where I wanted to be. And although your employer might not like this, for me it meant jumping around until I found the thing that made me happy.

It can happen….I am living proof!

Check out Kelly’s post. How does it resonate with you?

I was going to do the last class on a Saturday in GMT http://dddcqrs3.eventbrite.com but a bunch of North Americaners were saying they would like a weekend version as well in their timezones. As such I put up one for Saturday that people can register for http://usacqrsddd.eventbrite.com/ 

The material is the same as previously just in a weekend time format.

Material covered includes:

• CQRS
• Event Sourcing
• Building an Event Store
• How the domain changes
• Evolving an existing architecture
• Testing
• Eventual Consistency
• Versioning

The format of this is meant to be open to the community. As such the only form of payment is donations.

Still neck-deep in start-up mode these days. I still don’t feel any different than I did before we started this venture whole hog in January but that might be because I was prepared for it. Since the baby came in November, I was already familiar with the lack of sleep, persistent anxiety, and the constant mood swings between “this is the greatest feeling ever” and “what the &*%$ was I thinking?!?” so it was a smooth segue to entrepreneurship.

In any case, I’ve waxed somewhat comprehensible on the topic of working remotely fairly often. In most cases, I worked on a team where I was the only remote person. Nowadays, I’m part of a start-up where the entire dev team is scattered. And long-time readers will be horrified to hear that I’m the one in charge.

It’s been a learning process, to be sure, and continues to be. We’ve adopted procedures that I increasingly refer to as “spry” in that they help us work faster. But I’m scared to call it Agile or Lean because my knowledge of either of those processes is limited to what I read about it in the little-known Bazooka Joe comic series on the subject. So in order to avoid much “tut-tut”-ing from people on Twitter, I just say “this is what we do; if it follows a formal methodology, that’s not my fault.”

To that end, I’m going to fly directly in the face of the “favour individuals and interaction over processes and tools” part of Agile and post on a few of the tools we’ve been using to keep us organized. Or, more accurately, keep us from becoming too disorganized. In this episode:

AgileZen

When we first started our venture, my partner had everything in BaseCamp. It worked well enough. The Writeboard feature was useful and it delivers on its promise of keeping communication centralized. But we were using a task list to manage the features we wanted to put in and it very quickly became unwieldy for that. Not to knock BaseCamp. It is in the realm of software we want to emulate in that it’s great at what it does and makes no apologies if it doesn’t work outside of those boundaries.

The two obvious choices in my mind were LeanKitKanban and AgileZen. We took a look at both tools and decided on AgileZen because of its simplicity. We also took a very brief look at AxoSoft but it was deemed overkill for our needs. The pricing page alone doesn’t exactly inspire confidence in the whole “keep it simple” thing we were trying to strive for.

Likes

There’s plenty to like about AgileZen. The visual nature of it is well-suited for a remote team for daily stand-ups. Our process is generally to leave the board as-is for most of the day and move things around just at the stand-up. Helps give a sense of progress that was accomplished the previous day.

AgileZen follows a growing trend with software: less is more. This is an approach espoused by 37Signals and it’s one we’re very much on board with in our own product. So it gives us a nice warm fuzzy feeling to use a product that aligns with our own philosophy.

The overall user experience shows great attention to detail. It wasn’t until very recently, after having used the product for months, that I noticed the Add Story “form” had no labels. Instead, all the fields have watermarks indicating what to put in the field.

Because of this, the overall feature set doesn’t really give a decent picture of the application. Yes, you can colour code stories and add tags and, more recently, filter based on search criteria. None of these individually make AgileZen better than competitors. It’s more the way they all seem to work as a cohesive product.

Wish List

There is but one single feature that I think is the sole major gap in the product: a full API. There is a REST API to query things but nothing to write information back. (At least not yet; the feature is in the works.) This is something that would have *really* come in handy when we migrated all our tasks out of BaseCamp. But even now, stories come from other sources and almost all of the other products we use (e.g. BitBucket, ZenDesk) have an API so it would be pretty trivial to write a little utility to, say, take a ticket from ZenDesk and create a story from it in AgileZen. (And as I noted on Twitter, no prizes for what such a utility would be called.)

Also, as much as I marvel at the user experience, there are some minor quirks. The drag and drop isn’t infallible. There have been many cases when I just wish there was an arrow icon that I could click to move it to the next column without me having to fight with drag-n-drop, especially on my laptop where I’m forced to use the touchpad.

Couple of other annoyances are very likely low priority due to the potential number of people it affects. First, I use the Vimium Chrome extension which makes it easy to navigate a page with the keyboard. But some areas of the app aren’t “clickable” with this extension. Second, the fact that it kicks me out after I log in from another computer. I use AgileZen from at least two different virtual machines on any given day so I often have to re-login. This has been alleviated thanks to Vimium and AutoHotkey using a similar technique I used to migrate the stories out of Basecamp. Now, I just press Win+a to log in to AgileZen.

I hesitate to mention this possible feature but it’s something that would come in handy for us: threaded discussions attached to a story. The reason I hesitate is because, frankly, if I were on the AgileZen team and someone asked for this, my answer would be a flamboyant “suck it, user!”. It pretty much flies in the face of everything AgileZen is trying to achieve with its feature set. Internally, when we discuss a feature like this that would very obviously make the app overly complicated, we usually say discard the idea and say, “Yeah, there’s no need to Microsoft the app.” That’s how I predict the conversation would go with threaded discussions in AgileZen.

This list is kind of a double-edged sword because as I mentioned, any new features threatens the “keep it simple” vibe the app has at the moment. That said, Nate and Niki are smart people and I trust they will find a way to give me what I need, even if it’s not exactly what I asked for.

Wrap-up

Besides using the software, AgileZen has a certain quality that we are actively aspiring to achieve. It has the type of user experience that produces an emotional response. That response is almost universally positive so much so that even though it has a few quirks, I love using it. It’s something we use everyday and coupled with a decent screen-sharing tool (which is coming up next), it’s been a valuable tool for our remote team.

Code evolution and code maintenance are some of the most prominent characteristics of software engineering. Visual Studio doesn’t deal directly with code evolution. To explore code changes one needs to plug a Source Control Manager (like TFS) to Visual Studio. Source Control Manager can tell that a source file has been changed and it can trace text change history. But there is an impedance mismatch : Source Control Manager deals with text files while Visual Studio deals with code. Source Control Manager won’t make subtle difference between comments change, code in method refactored, type added, method visibility change or field removed.

NDepend comes with some advanced code evolution and code diff features since mid 2007. One of the coolest results of NDepend v3 integrated in Visual Studio 2010, 2008 and 2005, is that now, developer benefit from several fine-grained code diff features, one click away from their natural developing environment. Let‘s expose 3 typical code diff scenarios.

Code Diff from within Code Source

Since NDepend recognizes the code element currently edited in source, the user can, for instance, right click a method and see if it has been changed. If the method code has indeed been changed, several options appear on the right click NDepend menu. One of the option is Compare Older and Newer version of source file, which leads to opening a diff textual application to see changes



Here is for example,the  Araxis Merge textual diff tool opened from clicking Compare Older and newer version of source file on the NUnit v2.4.8 vs. v2.5.3 method IsPlatformSupported().



Look back in the first screenshot above, and you’ll see several other interesting options, including Open my declaration in older source code and Compare older and newer version disassembled with Reflector. With this last option (with Reflector) the language used for disassembling can be chosen (choosing IL is especially interesting). Also, comparing disassembled version can work, even from just 2 different versions of an assembly, the source code is not required.

Let’s precise that not only method changes are taken account but namespace, type and field changes as well. For example, one can right click a namespace source code declaration and ask for types or methods of the namespace that has been refactored or added.

At this point you might wonder how NDepend/Visual Studio knows about the previous version of the code to compare against to (what is called the Baseline for Comparison). This concept is explained later in this post, let’s jump to the second VS code diff scenario.

Code Diff from within Solution Exlorer or Solution Navigator

As we just saw, there is an NDepend menu in source code right-click menu, but there is also an NDepend menu in Solution Explorer (and in Solution Navigator) right-click menu. So for example one can right-click a project in the Solution Explorer and ask for Select Methods (of the project) where Code was Changed.

This will result in a CQL query generated that Select Methods (of the project) where Code was Changed. Naturally, for each method listed, one can jump straight to source code declaration or ask for diff.



One point interesting, is that NDepend comes with a heuristic to infer namespace from Solution Explorer folders. Indeed, it is a popular good practice to organize source code in a folders hierarchy that mirrors the namespace hierarchy. So right-clicking a folder can result in asking for code changes in a namespace, a narrowed scope than project.

Code Change Review through Global Code Diff

The third scenario to expose, is Global Code Diff. NDepend has its own global Visual Studio menu that comes with a sub-menu Compare.



This sub-menu Compare provides several options, including Search some particular Code Element changed. This option opens the NDepend Search panel with the option Search Change. The Search Change Panel is actually just a CQL query generator related to Code change. For example, in the screenshot below, we can see that asking for Method + Change + Code was Changed or was Added generates the CQL query:  SELECT METHODS WHERE CodeWasChanged OR WasAdded

The result shown is ideal to do efficient Code Change Review: not only all code changes are nicely organized at a glance, but for each methods refactored, the developer is just one click away to observe diff in source files.

The Search Change Panel’s options offers various possibilities to explore diff, including searching for code elements where code and only code (not comment) was changed or where comment and only comment was changed, where visibility was changed, where was added or removed etc….

A bonus option is to search diff in tier code, like asking for which library types is not used anymore for example.

Typically, developers like to write tests for testing automatically refactored and new code. Another cool bonus option is to search for diff coupled with code coverage by tests ratio, like asking for methods where code was changed (i.e refactored methods) and where it is not actually properly covered by tests.

Defining the Baseline for Comparison

Earlier in the post we introduced the concept of Baseline for Comparison.  This represents the previous snapshot version of the code base against which, the comparison is done. Typically, the Baseline for Comparison represents the latest version of the code in production.

The Baseline for Comparison can be specified through the menu: Visual Studio > NDepend > Compare > Define the project’s Baseline for Comparison. The dialog below appears and let chooses through different options to define the baseline. The Baseline for Comparison option is then persisted in the NDepend project file.



To harness all the diff capabilities we presented, it is important that the Baseline for Comparison points to a previous analysis result with .NET assemblies, PDB files and source code, all available somewhere on the local machine. If PDB files point to a different root folders (typically because .NET assemblies have been created on a different machine) NDepend comes with a source file rebasing option. If older version of source files is not available, it is still possible to query for what was changed, but of course, this won’t be possible to compare the missing older version of source files with the newer version. However, the option Compare older and newer version disassembled with Reflector will still be available as long as the older version of the .NET assemblies is available (no matter if PDB files and source files are absent).

Finally, any diff tools can be plugged through the menu Visual Studio > NDepend > Compare > Source Files Compare Tools (this is something explained in details in the post Diff tools).

Enjoy live code diffing in Visual Studio!

In Improving Your Audio: Hardware Edition, I focused on the importance of good audio hardware. No amount of post-processing is going to turn poor raw audio into a listenable podcast/webcast/screencast. It would be like trying to print a high resolution image from a grainy scan. Sure you can interpolate pixels to clean up the graininess, but you’re not going to make detail magically appear that wasn’t in the original scan. The same is true with audio. You can clean up bad audio by removing pops and hisses, but you’re not going to make good sound magically appear from poor quality raw audio.

One inexpensive piece of equipment that will save you a lot of retakes and clean-up is the pop filter. A pop filter is a thin screen of fabric that sits between you and your microphone and will set you back about $20. The pop filter is the black circle on the gooseneck in the image on right. (Image used with permission under Creative Common Attribution 2.0.) A pop filter prevents “p” and “t” sounds from making “popping” sounds in your audio. Without a pop filter, you end up with audio like this:

I’ve recorded the same phrase at the same distance from the microphone, but now with a pop filter between me and the mic:

The popping is caused by the rapid burst of air overloading the input capacity of the microphone, which results in clipping. You can see it if we look at the waveform of the raw audio.

Notice how the audio in the top recording is clipped where the microphone is overloaded. (+/- 1.0 is 100% input in Audacity.)

Most headset microphones like the LifeChat LX-3000 have a wind shield, which performs the same function as a pop filter. A wind shield is a fancy term for that piece of foam on the actual microphone. The main disadvantage of a wind shield over a pop filter is that wind shields “colour” the audio more. A good pop filter is acoustically neutral, which means that your audio sounds the same with and without the pop filter – it only eliminates the popping from “p” and “t” sounds. Also remember to attach your pop filter to your mic stand or boom and not directly to the microphone otherwise the microphone will pick up vibrations from the pop filter.

The moral of the story… If you’re going to spend money on good audio gear, don’t forget to buy a pop filter. The $20 it costs you will more than pay for itself in better audio quality and time saved in fewer edits and retakes.

Until next time, happy ‘casting!

All,

I've come to realize this week that I'm in need of an extended mental holiday from the absurd level of side project work that I've been doing for the past several years.  I have some work on FubuMVC coming up as part of a Dovetail project, but otherwise I think I'll try to recharge the batteries for a while and shut down the computer at night for awhile.

StoryTeller

As of this moment, I'm shelving StoryTeller altogether.  The 1.0 release was admittedly rushed for an early adopter and has been problematic to get up and running out in the wild.   I know that StoryTeller needs some project automated "warmup" infrastructure and a *lot* more documentation and blogging to make it more approachable, but I'm stretched too thin to do that work right now and there frankly isn't enough interest to justify the effort.  The code is up on GitHub if anybody wants to play with it -- but frankly, don't use it unless you're willing to get into the code and contribute pull requests back in.  At this juncture, I'm recommending that folks look at other alternatives like Cucumber and its .Net clones.  I'm still saying that FitNesse is nothing but a high friction tool that you're better off without.

StructureMap

StructureMap is a different story because people actually use it and depend on it.  StructureMap has some known weaknesses with its nested container support, the obsolete method noise to remove, people wanting more and more convention support, and better lifecycle support that I'd like to address with the 3.0 version I've had on hold for 3 months while I concentrated on Storyteller -- plus there's the endless whining on Twitter about the documentation being out of date.  It's been on my TODO list for 6 months to convert the documentation over to the Spree guides like the ones for FubuMVC for easier maintenance, but haven't had the bandwidth.  If you have too much time on your hands, I would dearly love to have a volunteer or two to help me refresh the docs infrastructure and get more examples in place, especially for the technologies that I don't use regularly like ASP.Net MVC, WCF, EF, Linq to Sql, and WPF.  I will get back to StructureMap work before summer is over, but I'm prioritizing my own mental health for awhile first.

Oh, and I'm swearing off Twitter again.  Nothing but snark, pessimism, and negativity there, especially from the .Net side of the fence.  Should improve with me in a timeout anyway;)

I'll see y'all later when things seem like fun again,

Jeremy

We’ve covered quite a bit in this series including how well the Reactive Extensions for JavaScript plays with other libraries and what integration points we have.  Our main thrust in providing this is that we don’t want to replace your library of choice, whether it be jQuery, Dojo, MooTools, YUI in addition to the server components of node.js.  Instead, we want to build bridges from them when you encounter the pains of asynchronous and event-based composition which frequently crop up in JavaScript applications both on the web and even on the server.  Case in point, let’s go over our latest integration with the Dojo Toolkit’s promises via the dojo.Deferred module.

Before we get started, let’s get caught up to where we are today:

• Introduction
• Creating Observables
• Creating Observers
• jQuery Integration
• Composing Callbacks
• From Blocking to Async
• Wikipedia Lookup
• Composing Deeper
• Bing Maps and Twitter Mashup
• Drag and Drop
• jQuery Live Event integration
• jQuery AJAX integration
• A Separation of Concerns
• Aggregates – Part 1 and Part 2
• Join Operators
• Going “Parallel” with ForkJoin
• Refactoring a Game
• Async Method Chaining
• Custom Schedulers
• Countdown Timers
• Wrapping the Dojo Toolkit
• MooTools Integration
• Conditional Operators
• Looping Operators
• Error Handling Part 1 and Part 2

Dojo Deferred

The Dojo dojo.Deferred module is a central piece of the Dojo Toolkit which serves as the basis for such things as AJAX calls like xhrGet among others.  At its core, it serves as a promises API which acts as a proxy for value which has not yet been calculated, and through this API, we have a clear separation of concerns which allows us to abstract the asynchronous behavior via callbacks into the promise itself, instead of the method signature.  The same could be said of the Reactive Extensions for JavaScript as well. 

Although dojo.Deferred has been around for a while, the newest release of the Dojo Toolkit has added another method to handle both the success and failure cases of a given computation, such as the following code.  Imagine we had an asynchronous function which returns to us a dojo.Deferred object, which does nothing more than delay by a second and return the value.

function someDeferredFunction() {
    var deferred = new dojo.Deferred();
    setTimeout(
        function() {
            deferred.callback(42); 
        }, 1000);
    return deferred;    
}

Then we can subscribe to both its success and failure cases by using the then function.

var deferred1 = someDeferredFunction();
deferred1.then(
    // Success
    function(value) {
        alert(value);
    },
    // Failure
    function(error) {
        alert(error);
    });

We can see a bit of overlap between the ideas here and the heart of the Reactive Extensions for JavaScript.  The difference between the two is that this API treats asynchronous calls as one shot resumable methods whereas RxJS treats these as observable sequences.  So, how could we take advantage of this API and build a bridge between the two libraries?

From Deferred to Observable

The first order of business is to see how we can take a given dojo.Deferred and turn it into an Observable sequence yielding a single value.   Using the Reactive Extensions for JavaScript with the Dojo support added via the “rx.dojo.js” script, we can seamless leap between the two by calling the asObservable prototype method we added.

var deferred2 = someDeferredFunction();
deferred2.asObservable()
    .Select(function(value) { return value * value; })
    .Subscribe(
        function(value) {
            alert(value);
        });

In this example, we took the someDeferredFunction from above and then called the asObservable method, then we’re free to do such things as projections via Select in which we square the returned value and then subscribe to the result via Subscribe.  To show you how we did it, let’s walk through the code below.  First, we create an AsyncSubject which represents and asynchronous operation which accepts only one value and then caches it for all future observations.  Then we can call the then function and in the success case, we call OnNext with the result and OnCompleted to indicate we are finished, and in the failure case, we send the error on via OnError.  FInally we return the subject, thus completing our bridge.

dojo.Deferred.prototype.asObservable = function () {
    var subject = new Rx.AsyncSubject();
    
    this.then(
        // Success
        function (value) {
            subject.OnNext(value);
            subject.OnCompleted();
        },
        // Failure
        function (error) {
            subject.OnError(error);
        });
        
    return subject;
};

But, what about the other way around?  Can we go from an Observable sequence to a dojo.Deferred?

From Observable to Deferred

Because we feel like building bridges, we want to give the ability for traffic to go both ways.  For example, we could have a given AsyncSubject that we want to use with our existing Dojo functionality.  Imagine we had an Observable sequence that waits for a second and then yields the value via an AsyncSubject.

function someAsyncSubjectFunction() {
    var subject = new Rx.AsyncSubject();
    
    setTimeout(
        function() {
            subject.OnNext(42);
            subject.OnCompleted();
        },
        1000);
        
    return subject;
}

Now, we might have some processing logic written in Dojo, so we want you to take full advantage of that.  Via the AsDeferred method provided to us through the “rx.dojo.js” script, we build that bridge between the two libraries.  After calling AsDeferred, we can then use it as a regular dojo.Deferred object.  So, now we can call then passing in the two continuations for the success and failure cases.

var subject = someAsyncSubjectFunction();
subject
    .AsDeferred()
    .then(
        function (value) {
            alert(value);
        },
        function (error) {
            alert(error);
        });

Now, how did we make this happen?  We’ll need to extend the AsyncSubject with our AsDeferred method.  Inside, we’ll create a new dojo.Deferred which will handle the values from our AsyncSubject.  Next, we’ll subscribe to our subject via the Subscribe method, calling the callback method with our result on success, and errback on the failure.  Finally, we return the deferred which completes our bridge.

Rx.AsyncSubject.prototype.AsDeferred = function () {
   var deferred = new dojo.Deferred();
   
    this.Subscribe(
        function (value) {
            deferred.callback(value);
        },
        function (error) {
            deferred.errback(error);
        });
        
    return deferred;
}

And there you have it, our bridge between the Dojo Toolkit and the Reactive Extensions for JavaScript, which allows you to have the best of both worlds.

Conclusion

Dealing with asynchronous programming has been in the forefront of many minds in the JavaScript community.  At JSConf, there were several examples of frameworks trying to get around the idea of callbacks and instead lean more towards composition.  By utilizing the Reactive Extensions for JavaScript, we’re able to compose together asynchronous and event-based operations together and transform the results in interesting ways.

The Reactive Extensions isn’t meant to be the only solution in your toolkit, but instead you can continue to use the JavaScript library of your choice, and when you run into callback hell and composition issues, that’s where RxJS shines.  To that end, we’ve provided a lot of bridges between various libraries such as the following, and more to come:

• jQuery
• Dojo Toolkit
• MooTools
• Prototype
• YUI3
• ExtJS
• Raphael
• Bing Maps
• Bing Translator
• Google Maps
• Google Translate
• Node.js
• and so on…

So with that, download it, and give the team feedback!

This post comes from a discussion I had with Clemens Vasters at NDC but is a much bigger discussion than the context we had it in which was Azure specific. An exaple of where this also comes into play is with the REST crowd who also want all idempotent operations.

Sure distributed transactions are evil ... until well they aren't. There is a trade off between capital investment and *insert whatever reason you don't like distributed transactions*. First let's look at the alternative.

To briefly cover idempotency from Gregor

"In the world of distributed systems, idempotency translates to the fact that an operation can be invoked repeatedly without changing the result."

This is a huge benefit because in failure conditions etc, I can just send you the same message multiple times and it won't affect you in a bad way. As an example I peek a queue and process the message writing something. Then suddenly my power goes out before I have actually removed the item off the queue. When I start up, I can just re-process the message (it won't cause destructive change).

Many operations are idempotent to start with. Consider a read or the setting of a property. Other options are not idempotent and need code to be written to make them idempotent as an example charging someone's credit card what happens if you get that message twice, ouch. The trick here is the operations that are not naturally idempotent, they require us to write code to make them idempotent. That code has a cost associated with it in terms of creation, maintenance, testing, and conceptually.

Clemens gave an example of a system he worked on to illustrate his point, a credit card processing system.

A credit card processing system has a very limited number of possible transaction types. For sake of discussion let's say that there are 15 and they are called very often, each has a large amount of logic associated with each transaction type (a similar example might be a trading system, not a lot of command types but a lot of logic associated with each). In these cases the idempotency is not a big deal because the amount of code being added is very small for it as a percentage of the total code.

This is very different than a stereotypical business application where there will orders of magnitude more transaction types and they will be often called rarely with small amounts of logic behind them ... for sake of discussion let's say closer to 1500 types with the detection code being vastly larger as a percentage (I have seen business systems where there was more code to ensure the idempotency than to perform the actual operation).

We can make everything idempotent but we will have a very different cost associated with it in the stereotypical business system than with the first system. Out of our 1500 operations how many of them are not naturally idempotent? How much code will we have to write in order to make them idempotent (quite likely ALOT!). Looked at as a percentage of the overall code it will be much higher than in the first example as there is much less code per transaction type on average.

Now what are we gaining by this overhead in order to have all idempotent operations? Scalability and Availability are two things that come immediately to mind. It is very hard to scale distributed transactions to an extremely high level (think Azure scale), many other trade offs start to come in at this level of scaling.

Are you scaling to Azure level? What is the ROI of that code you wrote to make things idempotent for the business system? This is not to say to use distributed transactions haphazardly (you always want to limit their usage as much as possible) but to point out that there are certainly situations where they may be favorable, there is a tradeoff involved. Again this is not to pick on Azure alone, REST also does this.

It is important to consider this tradeoff when you decide to say put an application onto Azure. How well do their non-functional requirements as a platform meet yours as an application? What types of decisions like this have they forced upon you and what will be the long term cost of those decisions.

I was just writing an email to the cqrs group http://groups.google.com/group/dddcqrs and figured it might be useful to put it up here as well as its a very common question I get.

The initial question was

Here's the example I'm using:
A system that handles user registration for 2 million+ active users. These users should be able to login with their email address and password. They should also be able to change their associated email.

I have the following design: 

Entities:
User(Guid Id, string email, string hashedPassword)

Events:
UserRegistered(Guid Id, string email, string hashedPassword)
UserEmailAddressUpdated(Guid Id, string email)

Commands:
RegisterUser(Guid Id, string email, string hashedPassword)
UpdateEmailAddressForUser(Guid Id, string email)

ViewCaches:
RegisteredEmailAddresses(emailAddress) - Used for client side validation on email prior to sending a RegisterUser command

When processing a RegisterUser command, I need to validate that no other user has registered with that email. How can I do that without loading every user in the system? I could use a view cache like the client side, but then I would have business logic outside of my domain. Any suggestions?

This is a very common question. There were many responses with various suggestions. Mine follows.

I am just replying to the last one on the list after reading through.

To me the most important concepts have been completely missed in this
thread and they are a big part of why eventual consistency is so cool
(it makes you think about things).

*What is the business impact of having a failure*

This is the key question we need to ask and it will drive our solution
in how to handle this issue as we have many choices of varying degrees
of difficulty.

Most of the time the business impact of such a failure is low and the
probability of it happening is low. If we query the eventually
consistent store at the time of submission (either from client or from
server as this is a big part of how one-way commands work) then our
probability of receiving a duplication is directly calculable based on
the amount of eventual consistency. We can drive this probability down
by lowering our SLA very often this is enough.

We can detect asynchronously if we broke our invariant. Imagine an
eventhandler that inserts into a table with a constraint. If it gets
an exception, we broke the constraint (note this is not really the
"read model" but the same db can be used if convenient, it is
important to note the distinction as if we scale to have 5 read models
we don't have 5 of these ...).

What do we do if we break the constraint? We need to come back to that
business impact statement above. For most circumstances, just raising
an alert to an admin etc is enough, these things are very low
probability of happening and are often not worth the time/cost of
implementing automatic recovery. Just imagine 1 username create out of
1,000,000 fails this way. How long would it take to automate the
process of handling the situation? Consider discussions with domain
experts etc. 5 minutes of admin time once a year is much better ROI in
most of these situations than a week of developer time to automate.

Continuing along it has now been decided that this has large enough
impact that it should be automated. The said process that finds the
duplicates could either raise an event DuplicateUsernameDetected or
directly call a command ResolveDuplicateUsername (which involves more
discussion). It is important to note that in either of these cases we
are discussing the "What" not the "How" it would never issue a command
"DeleteUser" etc, how to handle these situations is core domain logic
and should be modeled within the domain. In the username example
perhaps ResolveDuplicateUsername marks the user as not being able to
login (and as a duplicate) and it sends an email to the user saying
"Hey we screwed up but its your lucky day! you get to create a new
username ..."

But even after all of this if from a business perspective the impact
is too high we can still make things consistent. We could drop in a
service to the domain that deals with a consistent set. This would of
course be the last resort as its the most complicated of these
solutions and brings with it many limiting factors in terms of our
architecture.

Udi had a great example of this in his explanation of 1-way commands.
It was an ATM that would spit out money having only read your balance
from an eventually consistent read model. The reason this can work is
that from a business perspective the risk is low (and it is built into
the business model itself). You have a bank account with me, I know
your SS# and all of your information. For people who overdraw their
accounts I will recover atleast 90% of the money that has been
overdrawn. On top of that I charge a fee for each overdraw that
occurs. For these reasons the business impact of such a problem is
low.

To sum up I just want to reiterate that this is a *good* thing.
Eventual consistency is forcing us to learn more about our domain. It
is forcing us to ask questions that are otherwise often not asked.

Consistency is over-rated.