Http Caching is complicated for everyone

As you may know, one of the talks I’m giving this year is HTTP caching 101.

I feel it’s important to educate everyone on HTTP. One of the interesting things is the fact that the spec names things in a very, well, strange way.

Take my good friend Ayende that complains about Silverlight not respecting the HTTP specification. Now I think everyone will agree that Oren is a brilliant person, so if he gets it wrong, it’s probably the spec that sucks. Point is, silverlight is not necessarily wrong in this case.

See, a resource representation (the thing that is controlled by caching instructions) can be in three states: Up-to-date, fresh and stale.

Up-to date representations are the ones that we know are current according to the origin server.

When a representation is being sent back with no expiry and no caching instructions, it’s allowed (and encouraged) to cache it for however long an implementation may decide, it’s left up to the developers. The spec encourages using heuristics for this. If you want something more reliable, you should provide a max-age or an Expires (or ideally, both).

Now, as long as the representation hasn’t expired, it’s considered by caches (proxies or your browser cache) as fresh.

Now that’s where things become confusing. When you send back a must-revalidate with your response, it doesn’t mean that the client needs to revalidate at all times with the server. It means that a client needs to revalidate once the representation is stale, aka it has gone over it’s expiration date. In the case ayende shows here, the client makes the decision that the data was cacheable previously (I’d think that the response would’ve had a Last-Modified so heuristics could kick in), and so doesn’t see the point of asking the server to validate a fresh cache entry. This is why when Oren added the Expires header in the past, the entry became stale on arrival, making must-revalidate work as expected.

So what if you want the client to revalidate always before serving a cached entry? In that case, you can send a no-cache, which of course doesn’t mean do not cache. It means that the client needs to revalidate both fresh and stale entries, and then may use the cached copy if it’s up to date.

So to sum up: no-cache doesn’t mean do not cache, must-revalidate doesn’t mean always revalidate, most people get http caching wrong because it is not intuitive Thankfully, if you come to my talk, or my ReST training course, or use the openrasta-caching plugin, you won’t have to care much about those problems.

Posted in Uncategorized | 4 Comments

Service Bus support in azure npm 0.5.2

This just in:

Andre (my team) just published the latest version of our node azure module – 0.5.2:

Primary changes:

  • Azure Service Bus support!
  • Node.exe version requirement lowered to raise compatibility. This should enable using our SDK on Heroku and should make Juan Pablo Garcia and Matias pretty happy.
  • Multiple Bugfixes
  • Storage Services UT run against a mock server which should make tests run much faster. (just “npm test”)

Service Bus

The Service Bus provides secure messaging and relay capabilities that enable building distributed and loosely-coupled applications in the cloud, as well hybrid application across both private and public clouds. It supports multiple messaging protocols and patterns and handles delivery assurance, reliable messaging and scale for your applications. The Service Bus is a managed service that is operated by Microsoft and has a 99.9% monthly SLA. more here

In our latest update, you can now access Service Bus right from within your node.js applications! Below are snippets indicating usage.

Here’s an example of using the apis to create a queue

var serviceBusService = azure.createServiceBusService();
serviceBusService.createQueueIfNotExists('taskqueue', function(error, created){
    if(!created){
        // Queue exists
    }
});
view raw gistfile1.js This Gist brought to you by GitHub.

Here is how you send a message with a specific topic

var serviceBusService = azure.createServiceBusService();
serviceBusService.sendTopicMessage('taskdiscussion', 'Hello world!', function(error){
    if(!error){
        // Message sent
    }
});
view raw gistfile1.js This Gist brought to you by GitHub.

And finally this is how you poll the queue for a message.

var serviceBusService = azure.createServiceBusService(),
    topic = 'taskdiscussion',
    subscription = 'client1';

serviceBusService.createSubscription(topic, subscription, function(error1){
    if(!error1){
        // Subscription created

        serviceBusService.receiveSubscriptionMessage(topic, subscription, function(error2, serverMessage){
            if(!error2){
                // Process message
            }
        });
     }
});
view raw gistfile1.js This Gist brought to you by GitHub.

Be sure to also check out our unit tests for more samples. We’ll also have content on the dev portal for Service Bus very shortly.

We would like to thank the community and our advisors for the feedback and suggestions so far, which we have tried to accommodate as much possible for this release.

We’d also like to thank the community contributors that participated (Einar in particular a big thanks) in this release. We are looking forward to more contributions in the future :)

As to the road ahead, well here’s a little teaser…..

  • Service runtime –> Allows you to obtain and control the information on the actual machine your app is running in.
  • Service management and deployment –> Manage and deploy services on any platform using node cli tools.

Yes, we are still having lots of fun!

Posted in azure, node, node.js | 2 Comments

ASP.NET Connections – JavaScript Testing

I am honored to be presenting at ASP.NET Connections, March 26-29 in Las Vegas. One of the sessions I’m presenting is on the topic of JavaScript Testing. If you are writing web applications, JavaScript is likely as significant, if not more so, than your native C#/VB code. And yet, it  is often treated differently from a testing perspective. How does one organize JavaScript to make it testable? How do the SOLID principles apply? What tools can I use to test JavaScript? These questions and others will be addressed through practical code examples on how to use testing frameworks like QUnit and how those tools can be integrated into Visual Studio.

To more illustrate what this session will cover, consider the this hypothetical: we need to display address components in a pre-defined section of a page.

The end result might look like this:

The following code drives the address display:

$.getJSON("data/data.json", function (data) {}).success(function (data) {
		  $("#target").empty();
		  var items = [];
		  $.each(data, function (key, val) {
		    items.push('</pre>
<ul>
	<li id="' + key + '">' + val + '</li>
</ul>
<pre>
');
		  });

$('', {
		    html: items.join('')
		  }).appendTo('#target');

		})

		.error(function (error) {
		  $("#target").empty().append("

An error was encountered...

");
		})

		.complete(function (data) {
		  $("#target").removeClass('ajaxLoad');
		});

Fairly straightforward code: Data is acquired via getJSON and displayed. jQuery I used to drive the process. The code works, but there are problems. Recalling what we have learned from SOLID, DRY, etc. – we quickly learn that the code not maintainable, re-useable and it’s marginally testable. About the only thing that can be tested here is the end result. As to the target, were the proper classes assigned and is there an un ordered list with data? In other words, all we can really test are the side-effects of the JavaScript code, not the code itself. Looking at the code, there are several distinct things happening. First, there is the application of the ajaxLoad class. Second, there is the data acquisition process itself. Third, there is the application of the data. There are also operations to handle error conditions and when the ajax process has completed. Geographically, this code is embedded in an html document. Ideally, this code would be re-factored into its own js file. Going back to the distinct operations, each of these operations should be testable, and more specifically, unit testable. Ideally, we would like to verify our code with a test like this:

function testfixture() {
    module("When retrieving address data");
	//Arrange
	var html = $('</pre>
<div id="target"></div>
<pre>')
   //Act
   ApplySomeOperation(html);

    test("Given that the data has \
	  been applied to a pre-defined div",
	   function () {
	   expect(1);

	   var ul = $(html).find("ul");

       //Assert
	   equal(ul.length, 1, "The ul tag is present.");
	});
}

The code, as written, does not support a unit test like this. The code is essentially a big ball of mud. Again, the code “works”, but in our business, working is not enough. With a little re-factoring, we can get where we need to be.

This is what we end up with:

function removeStartMsg(target) {
   $(target).removeClass('ajaxLoad');
}
function displayStartMsg(target) {
   $(target)
      .empty()
      .addClass('ajaxLoad')
      .append("

One moment while your data is \
	    being retrieved...

");
}
function createAddressHtml(data) {
   var items = [];
      $.each(data, function(key, val) {
	     items
		   .push('</pre>
<ul>
	<li id="' + key + '">'
 + val + '</li>
</ul>
<pre>
');
	  });

return $('', {
		html: items.join('')
	   })
}
function displayAddress(data,target) {
   target.empty().html(createAddressHtml(data));
}
function getAddress(id,target) {
	var complete =  function(data) {
	    removeStartMsg(target);
		$(target).removeClass('ajaxLoad');
	};

	var success = function(data) {
		displayAddress(data,target)
	};

	var error = function(error) {
	   $(target).empty().append("
An error was encountered...

");
	}
	displayStartMsg(target);
	getData(id,"data/data.json",success,error,complete);
}
function getData(id,source,successCallBack,errorCallBack,completeCallBack) {
   $.getJSON(source, {id: id},
       function (data) {}).success(function (data) {
	   successCallBack(data)
   })
   .error(function (error) {
       errorCallBack(error)
   })
   .complete(function (data) {
   	   completeCallBack(data)
	});
}

The idea is that each discrete operation has its own function. Thus – we adhere to the single responsibility principle. Also note that no function reaches into the DOM. Instead, the object it is to act on is handed to it. The code in the page reduces to this:

<script type="text/javascript">
   getAddress(0,$("#target"));
</script>

For testing purposes, the json source is a static file on IIS. With little effort however, the source could easily be any http endpoint. With the refactored code, we can write more granular tests.

For example:

function testfixture() {
    module("When retrieving address data");
	//Arrange
	var html = $('<div id="target"></div>')
   //Act

   getAddress(0,html);
   stop(2); // allow async processes to run

    test("Given that the data has \
	  been applied to a pre-defined div",
	   function () {
	   expect(1);

	   var ul = html.find("ul");

       //Assert
	   equal(ul.length, 1, "The ul tag is present.");
	});

    test("Given that the data has \
	  been applied to a pre-defined div",
	   function () {
	   expect(1);
	   var expectedHtml = "<ul><li id=\"company\">Microsoft Corporation</li>
                <li id=\"Address\">One Microsoft Way</li><li id=\"City\">Redmond</li>
                <li id=\"State\">WA</li><li id=\"Zip\">98052</li></ul>"
       //Assert
	   equal(html.html(), expectedHtml, "The html markup is correct.");
	});
}

The following is the QUnit output:

Again, with the refactored code, we can test everything, at a unitary level, that is needed to support the display.

Posted in ASP.Net Connections, Javascript | Leave a comment

My appearance on Dot Net Rocks – SOPA

Recently, I was a guest on Dot Net Rocks. As you can see from the comments, this episode stirred the pot a bit. As I’ve always said, if you are not ticking off at least a few people – you are not trying hard enough! There is an update to this show that will be airing on 2/9. The original focus of the show was on SOPA – discussing what it is and perhaps more importantly, what it is not. During the show, I had added that one of the sources of intellectual property law are treaties that the United States ratifies. Seems like a non-consequential statement at the time. Turns out – it was not. For a few years, ACTA has been discussed  – but the topic has not received too much air play given that SOPA was in the news. Now that SOPA has been removed from consideration, ACTA has now entered the limelight.

I want to be clear here – I don’t agree with those that say that laws like SOPA and treaties like ACTA – subvert free speech. Stealing the intellectual property of others is not something that should be tolerated. That said, there were some troubling aspects in SOPA – mostly arising form the lack of due process. The major problem was that based on a mere allegation – a site could be shut down.  It remained to be seen how that law would have been enforced.

ACTA is another thing altogether in that it is a treaty. In my opinion, organizations like the Free Software Foundation (FSF) – completely mis-represent what ACTA is. If there is one thing about ACTA I think is worthy of criticism is the secrecy around the treaty negotiations.

In general, I’m a fan of the FSF. That said, the torch and pitchfork crowd goes a bit too far with their assertions. Here are the FSF’s points:

  1. It makes it more difficult to distribute free software: Without file sharing and P2P technologies like BitTorrent, distributing large amounts of free software becomes much harder, and more expensive. BitTorrent is a grassroots protocol that allows everyone to contribute to legally distributing free software.
  2. It will make it harder for users of free operating systems to play media: Consumers may no longer be able to buy media without DRM — and DRMed media cannot be played with free software.
  3. It increases the chances of getting your devices taken away: Portable media players that support free formats are less common than devices which support DRM, such as the iPod. Will this make them suspicious to border guards?
  4. It creates a culture of surveillance and suspicion, in which the freedom that is required to produce free software is seen as dangerous and threatening rather than creative, innovative, and exciting.

First off, folks need to read ACTA. It states pretty clearly that notions of due process and free speech must endure – that the manner of how ACTA is enforced must be consistient with those ideals. Put it this way, illegal activity occurs on cell phones and computers. Are we less likely to use those devices? Are those devices confiscated today? Of course not. File sharing technology supports legal activities. Indeed, it can support illegal activities as well. That however, does not mean those technologies will go away. This is where common sense and practicality have to kick in. For example – let’s say you in fact, have illegal copies of music on your iPod. How can you tell by looking at the device? More importantly, how could a customs tell? Do we really think this is going to be a priority? 

The fact is – pirated software, music and counterfeit goods are a problem – and it is well within the province of rights-holders to prosecute and protect their rights. That said, it cannot and should not happen at the expense of our rights and our rights to use technology that has legitimate purposes. I happen to think that the MPAA and RIAA, while they have some legitimate concerns, are also comprised of people that for the most part – are clueless on how to deal with these matters. If there is one point to take from the DNR episodes it is this – GET INVOLVED AND STAY INFORMED. If there is one suggestion I’d make to those who have to craft legislation and ratify treaties – it is this – G

 

 

Posted in ACTA, Dot Net Rocks, IP Law, SOPA, Uncategorized | Leave a comment

QA: A Hillbilly Love Story

The hillbilly turned forty last weekend! I’ve been anticipating this giddily because now, I can be crotchety and people will still find me adorable.

I hate QA. Hate, hate, hate, HATE, HATE, HATE it. I despise it so much, I think I might swear. Here goes….I f—, okay, I can’t do it but I still &*%$ hate it.

Side note: I know all QA people weren’t raised by trolls and/or gnomes but I’m going to generalize here. If you are a QA person and were raised by evil leprechauns, please don’t take offense.

As a hillbilly, I’m an innately optimistic and forgiving person. An ugly codebase to me is a sea of opportunity. Changing requirements? Ha! I can put my OCD up against any client. I can see the positive in almost anything. Except the emotional rollercoaster that is QA.

The end of an iteration is a time of celebration and relief for me. We’ve completed a significant piece of functionality on our application. We’ve followed best practices, left the code in a better state than when we started, and all our unit and UI tests are passing (more or less). Despite my past experience, I always feel great pride when I email our QA department:

Hey there, QA folkery! I come bearing tidings of great joy! We have a new version of the application for your revelry and astonishment! You will surely faint in awe at the sheer glory of it! I await your token sign-off. Acclaim and praise is optional (but recommended).

After that, I sit quietly for a few minutes to bask in my accomplishment of the last couple of weeks then dive into the next iteration, the previous one long since gone from the vestiges of my memory by that point.

The inevitable spartan response:

Please fix the following issues:

  • As per the spec, emails should be sent every hour, not daily
  • I got a 404 error when I went to the public booking URL
  • I know this wasn’t part of the iteration but we should display a message when someone confirms an appointment. Can you include this?
  • You spelled it “cancelled” in the settings screen but “canceled” on the calendar.

Let me know when a new version is up for testing.

By now you will not be surprised when I tell you that the QA process was invented by the Marquis de Sade.

QA goblins: surely you see my point, no? The iteration is over. Done. Finished. There’s no “D.S. al fine” at the end. In my head, I’ve moved on to new functionality. I’ve got tests written and prototypes completed. And you want me to forcibly re-enter the past?!? All the hard problems from that iteration have already been solved. I don’t care about your “cosmetics” or your “functionality”. That new feature you’ve requested? It’s KILLER! It’ll be a great asset to the app. When I get around to it. That 404 error? Just a configuration issue. It’ll be fixed when we deploy. And come on, spelling mistakes? Have you not been on Facebook lately?

In short: I. Don’t. Want. To. Do. It. Right. Now.

And I have to. Virtually everything brought up in a good QA process is valid from bugs to cosmetic issues to features that we need to include this iteration even though they weren’t in the list. Without them, we don’t have a usable product. Users will complain or worse, move on to something else. But my headspace is somewhere else. I’ve already compartmentalized the new features and have shifted my own personal Eye of Sauron on them. Now, there’s leakage that I thought I’d never have to deal with again.

Another psychological issue: I’ve poured everything I’ve got to deliver the best that I’ve got and it wasn’t good enough. It’s never good enough. I’ve just presented to you the end result of four years of university training and over a dozen years of experience as an “expert in the field”. And with a few phrases, you send me slinking back to my computer to do it over again.

It’s a little strange in some regard. I welcome criticism of my coding choices and style. I actively seek it out sometimes. Even code reviews can be made fun again. But something about the subtly adversarial nature of QA raises my dander. The fact that it’s so essential to quality and that I can’t think of an effective way around it doesn’t help…

So to sum up:

  • QA forces me to change my headspace. And I hate it for that.
  • QA points out my flaws as a developer. And I hate it for that.
  • QA is necessary and makes software better. And I hate it most of all for that.

To end on a more positive note, most QA people (including those that do it as part of other duties) I’ve worked with are extremely nice. There’s no accusatory tone in the responses, no blame on messing things up, and no sense of “you’ve *really* screwed things up this time.” In their minds, they’ve done what’s expected and like me, have made the product better for it.

But I still hate you.

Kyle the Unadulterated

Posted in Uncategorized | Tagged , | 9 Comments