Practical ReST – the book

I’ve been talking for years about writing a brain dump of what I know of ReST. The project has started, you can pledge to buy your copy today (in various editions) and get content as early as next month!

The response has been amazing, with three quarters of the minimum funding pledged in 48 hours. Of course, the more contributors the more time I can allocate to the book, the quicker the book will be in your hands, so don’t hesitate!

Writing a book in itself is a big challenge, and the scope is very ambitious. But it’s also a foray into self-publishing, and I expect it to be an amazing opportunity for learning.

Practical ReST – the class

I’ve been giving a ReST class for a while, but I’m rewriting it so it follows the same structure as the book, and make it open to people with many technological backgrounds.

First date should be in May in London, and I expect Warsaw, Vilnius and other locations to pop up this year.

If you want to see a workshop organised in your city (or your company), pop me an email.

I’m also launching a referral program *and* group rebates, so if you’re a training company and you want to add the course to your books, you can too.

Hanselminutes podcast

Scott was nice enough to have me on his show this week, talking about Practical ReST. It’s published and ready to be listened to, and while I was half-way through an asthma attack and having microphone problems, Scott saves the day by being a great host.

Cancer Research UK

You may remember that I asked for your contributions to help me raise money for cancer research UK. I can now announced that we raised more than 700£, and that both my objectives of non-drinking were achieved. I cannot thank you all enough for your generosity.

serialseb.com

It had to happen. I now have my own web site. You cannot imagine how difficult it is for me to put work and effort in talking about what I did rather than what I want to do. But you will find on there a link to all my previous talks, with videos and slides (where applicable), as well as where you’ll find me next, my future talks and where classes will be delivered next.

What next?

ReST will be my focus for the next few months. I have a couple of other projects up my sleeve, but this is enough announcements for now, don’t you think?

In the meantime, I’m giving my drinking money to charity by giving up alcohol, and I’ve set-up a challenge. If you help me raise £1,000 for Cancer Research UK, I will be shaving my head, and you’ll get a nice video of it too.

So if like me you feel that it’s your time to give back, but you’re attached to your Tony n Guy creation, or you just don’t want to give up that Pinot Gris, please help me raise funds by donating on http://www.justgiving.com/dryathlete-sebastien-lambla.

Over the years, the various component organizations within Microsoft have had various levels of involvement with Open Source, with different philosophies, and from what I have gathered over the years from post-conference drinks, internal conflicts that can be quite a challenge.

As this post is going to be long, I’m going to sum-up the problem, and you can go to the conclusion bit to read what I believe should be done differently.

Microsoft has no vision when it comes to Open-Source, no strategy and no leadership. Some groups are seen as more progressive, and do tend towards a better approach to OSS. Some people consider those steps as sufficient to stop pressuring Microsoft towards responsible OSS, and I disagree: it’s only some groups, and even within those groups, the advances are still very much one of exporter. Recognizing Microsoft as an OSS-friendly player is like recognizing the People’s Republic of China under the Reagan administration, it’s a step in the direction of normalization, not a recognition of the Human Right lifetime achievements of Mao Tsedung.

You know why I know Microsoft has no leadership and vision when it comes to Open-Source? Because I asked Steve Ballmer when he was in London, and he replied with this: “I don’t know, but we won’t impose any view on our divisions. I’ll come back to you by email though.” Steve, my email is still seb@serialseb.com and I’m looking forward to your views.

Where we come from

The current state of affairs is not bad, considering where Microsoft was years ago. We’re talking about a company that positioned itself as the defender of private code, vilifying OSS licenses, having no visible OSS project going and providing its value as the integrated stack: it’s all MS or none. We have certainly come a long way.

Opening up slowly

I think under the influence of Scott Guthrie, an internal change started deep within the asp.net team ranks, certainly a movement that resulted in the recent announcement of the opening up of the asp.net webstack as a true Open-Source project.

Now that’s commendable, and I applaud the move. Open-Sourcing and opening up to collaborative development for stuff you’ve built is great. Well done.

Not invented in Redmond syndrome

Let’s be clear. The goal of MVC was to build a better Monorail, as opposed to help Monorail productize their framework for inclusion in Visual Studio.

The exact same thing happened with Nuget. With multiple projects with different approaches already in the work, Microsoft started their NPack project (I seem to recall someone at Oredev mentioning it was originally built as part of the WebPages tooling). It wasn’t to be open-source to start with, hence why the existing OSS projects didn’t get the news until much later.

But, I hear you say, Nuget is an open-source project from Outercurve, and they combined their code with another project. The open-sourcing was decided very late in that process, once the investment in code was so large that the ship had left already and nothing would stop it. And on a side-note, there was no merging of the code, the Outercurve foundation’s press release was inaccurate, just look at the history.

What really happens when a new project gets produced by Microsoft? It sucks the air out of a lot of community projects, often including all the best ideas of said projects, and Microsoft gives nothing back. If you’re in that boat, you either redraw your plans based on their new announcements (usually rewriting on top of the new shiny stuff), or you dissolve your group and join their OSS project (the Nuget / Nubular case).

Certain projects (like nhibernate and nunit) continue to happily go forward, and maybe the OSS alternatives that are impacted by Microsoft entering a space just were not good enough. I don’t know, we’ll see in a few years.

Adoption of existing projects

Of course it can be said, and there again, I commend Microsoft for the move, that libraries such as jQuery, modernizr and now Newtonsoft.Json, are being used, supported by Microsoft and shipped alongside the product.

This is also a great departure from time immemorials. Makes no mistake however, Microsoft is being pushed by market pressure to integrate with those frameworks. The change is only that they’re abandoning slowly the idea that they’ll release until they gain marketshare, in those domains where Microsoft has historically been slow at releasing compared to alternative platforms.

This is only an opinion piece, but I’ll suggest that the move to gain web development share has pushed Microsoft towards dropping the projects they had and failed to make popular (the asp.net  AJAX stuff that got more or less killed when jQuery was adopted). And I think that’s great.

Pragmatism is also the reason why newtonsoft.json is being used by Web API. The current Json providers on .net are broken, there is no way to provide the customization people want, and fixing them cannot be done in the same release cycles as MVC, as some are part of the core framework (and may or may never be fixed, but certainly not in time for the next release).

It was not chosen because Microsoft wants to adopt more external OSS projects as their core vision. It was chosen because they couldn’t fix what was broken and rebuilding would have been silly.

On a side-note, twitter conversations started going in the direction of why Newtonsoft.Json was chosen over other libraries. I made the assertion that provided it’s an internal implementation with no exposed surface, the choice of *any* json lib would’ve been fine. ScottHa said it was because the library was the most popular, and testing was done on their part. That lead me to the assertion that if the only library under consideration was chosen for popularity, and there is no exposed surface of that lib on public APIs, the choice is one of brand-surfing and gives extra marketing points to their OSS credentials. If some of the API is exposed, it was probably a reasonable choice. I don’t care which library you chose, I care the process you used to chose which library and the reasoning behind it. I question (and by question, I mean discuss) the motives, not the solution.

[Edit: After looking for it (as the twiterverse didn’t reply to the specific of the discussion), one object from json.net *is* exposed on the JsonMediaTypeFormatter, so that is that and I’ll stick to my original analysis, it’s probably a reasonable choice.]

There is no vision, because there is no systemic approach to deciding between building and joining communities.

Asp.net Web API could have quite happily join existing OSS projects that dealt with this problem area years ago, but it wasn’t to be. Nuget funding could have been used on top of existing projects, instead of a pure creation, but it wasn’t to be either. What is the decision process behind the rebuild or join decision?

And can anyone that has had their project get the oxygen sucked out of, with nothing given back, be blamed for distrusting the company?

The corporation

When the Web API project was being kicked off, Glenn Block called me asking me if I thought he should take the job, and if he did, would I help. I told him what I’m telling you now: Yes, you can’t do as bad as the previous attempts, and I’ll happily help to make sure the architecture and model fits the reality of what developers want, we’ll all win. I helped because I was asked. Ideally I’d rather they had asked me to work on OpenRasta and create a new version with us, but it was not to be. In the end, the two frameworks may have very different code, but most of the concepts and conventions are exactly the same, so everyone will feel right at home on both platforms, and that’s in no small measure because the brilliant minds behind Web API respected HTTP as much as I and the other advisors did.

More recently, Drew Miller from the NuGet team has been very active in opening communication channels with OpenWrap, something that has not existed ever, and he should be applauded for it: I can now have a discussion about our interop story with someone that wants to make things better. Again, I’d rather the NuGet team had joined OpenWrap rather than communicate to us their imminent release, way too late to get any positive outcome, but at least it’s a step in the right direction.

Those are individuals, not the company, and out of all the voices out there, they are a minority. You cannot judge the whole company on specific individuals, and teams cannot use the excuse Microsoft is really many companies. That is certainly true, but completely irrelevant to external agents when judging how the company deals with OSS.

How to fix this

Three words, transparency, transparency, transparency.

When you have an idea for a product, and there are existing Open-Source projects out there that have adoption and could really help the resources of a big vendor, try and justify the choice.

If you’re going to rebuild, take the phone, call the leader, explain your plans, and don’t be a douche about it (aka telling someone “we realize we are stepping on toes here but we’re big so it’ll happen” is not particularly a great way to deliver the news). Chances are, they’ll probably understand the reasoning. ISVs have had that communication channel from Microsoft for years, and get to know about competing products way in advance, but it’s not the case for OSS projects.

Be open about the process afterwards too. The policy of not mentioning anyone, or pretending you cannot look at existing projects at all in public, would make anyone think you have something to hide. It probably is against some corporate policy somewhere, but make that change, you did do it about the no-OSS policy.

And finally, value the people on your platform, the loud mouths like me and the small guys that abandon their projects because you started yours. We are on your platform because we like it, and we want to make it better. And maybe if you’re going to step on people’s toes, try to find a band aid for them to sweeten the blow, and find a way to collaborate on something else with them. Don’t lose the people that have been developing the ideas you’re going to be building on, it’s not good business.

If these things had happened in the last few years, I wouldn’t distrust Microsoft-the-company as much as I do now. But the bridges that are being built by bright individuals inside the company do go in the right direction.

Is that going to be enough? I sincerely don’t know, but until we’re there, I’ll continue shouting at the top of my voice for Microsoft to become the business partner my OSS projects wants to be dealing with.

It’s called tough love, silly.

[Edit: added an additional paragraph to project competition]

[Edit 2: added a bit on the json.net choice]

I feel it’s important to educate everyone on HTTP. One of the interesting things is the fact that the spec names things in a very, well, strange way.

Take my good friend Ayende that complains about Silverlight not respecting the HTTP specification. Now I think everyone will agree that Oren is a brilliant person, so if he gets it wrong, it’s probably the spec that sucks. Point is, silverlight is not necessarily wrong in this case.

See, a resource representation (the thing that is controlled by caching instructions) can be in three states: Up-to-date, fresh and stale.

Up-to date representations are the ones that we know are current according to the origin server.

When a representation is being sent back with no expiry and no caching instructions, it’s allowed (and encouraged) to cache it for however long an implementation may decide, it’s left up to the developers. The spec encourages using heuristics for this. If you want something more reliable, you should provide a max-age or an Expires (or ideally, both).

Now, as long as the representation hasn’t expired, it’s considered by caches (proxies or your browser cache) as fresh.

Now that’s where things become confusing. When you send back a must-revalidate with your response, it doesn’t mean that the client needs to revalidate at all times with the server. It means that a client needs to revalidate once the representation is stale, aka it has gone over it’s expiration date. In the case ayende shows here, the client makes the decision that the data was cacheable previously (I’d think that the response would’ve had a Last-Modified so heuristics could kick in), and so doesn’t see the point of asking the server to validate a fresh cache entry. This is why when Oren added the Expires header in the past, the entry became stale on arrival, making must-revalidate work as expected.

So what if you want the client to revalidate always before serving a cached entry? In that case, you can send a no-cache, which of course doesn’t mean do not cache. It means that the client needs to revalidate both fresh and stale entries, and then may use the cached copy if it’s up to date.

So to sum up: no-cache doesn’t mean do not cache, must-revalidate doesn’t mean always revalidate, most people get http caching wrong because it is not intuitive Thankfully, if you come to my talk, or my ReST training course, or use the openrasta-caching plugin, you won’t have to care much about those problems.

OpenWrap has been there for a year and a half, and has some usage for advanced users wanting a nicely-architected package manager that doesn’t take dependencies on lots and lots of other technologies and doesn’t require VS.

And we have sub-projects, such as OpenFileSystem, an IO abstraction library that ought to  make using the file system easier, shipping with an in-memory version that should work the same way as your target operating system.

Now all those projects are at varying stages in their life, and the amount of work I used to give to only OpenRasta has been vastly disolved, which in turns means that some users are starting to get quite annoyed with our dreadful homepage and our lack of binary downloads.

Add to this the competitive pressure on OpenRasta introduced by Nancy and the future openrasta copycat WebAPI from the photocopier machine Microsoft, and you’ll understand that the current trend is not sustainable.

This year I’ve decided to simplify my life and delegate much more of my work to a community that has been so far very supportive but very unwilling to contribute anything back. I also want to go back to enjoying working on OSS software to build great solutions and not focus my energy on simply competing with any tool and framework microsoft invents to duplicate existing OSS solutions. I have a vision and I want to deliver on that, OpenRasta and OpenWrap are two sides of that, and I want to start making them work together the way I want without having to be dragged in competitive races with Microsoft’s marketing dollars.

So we had a planning meeting with some of the companies that use OpenRasta in anger in London and want to step up their contributions (7digital and huddle, we love you guys!), and decided that we were going to change how things are done in the OpenEverything projects.

Project organization

First and foremost, I’m going to contribute much less to the 2.1 branch of OpenRasta, and patches / merges / new builds are going to be handled by the contributors that are given write access to the git repository. This gives away a lot of my power.

I’m going to be focusing on the outstanding modules for OpenRasta that add a whole bunch of functionality that will be rolled-back in 3.0: the caching module (soon to arrive in 0.2 version), the new config API (with compact fluent API and convention-based config) and the dynamic / json out-of-the-box support. There’s also a new openrasta-diagnostics module that I’ve started building and want to get out there. I’ll also contribute some work towards the async support that has been requested by some.

OpenFileSystem 2.0 is going to be under Henrik Feldt supervision, and will be a merge of the amazing work he did on long-path and transactional file system support.

All of our versioning is going to be a strict application of http://semver.org 2.0 (which is now the only standard in OpenWrap), so that will solve any breaking changes being introduced without knowing.

Of course, OSS is not a democracy, so I keep a veto right on certain changes that would impact future versions, or architectural changes that do not fit with the vision I have for all those projects. This is give and take, I give some of the control of more than 4 years worth of my development time, but I will keep the main architect role for the time being.

As for releases, anytime someone with commit access wants to create a new release, the process is very simple: they email on the mailing list, and if there’s no majority opposition, we have a new build. Simples.

Orphaned projects

Sadly, we do have some orphaned projects in all this. Mostly, the container support for each of the IoC containers is not up-to-date for StructureMap (tests are broken) and Unity (I have no understanding of the code, so while I *think* I fixed the latest build, I can’t try it out). At this stage, anyone with a vested interest in those projects should start contributing and come help out, or risk that code not being maintained anymore.

OSS is not just take, it’s give too

That leaves a real question for those companies that have used OpenRasta to make money and provide a killer-architecture and have contributed absolutely nothing back since the beginning.

I’ll let you in on a little secret. I give the code away on everything, so it is a company’s legal right to take and not give back. As a side-effect, it is my complete right to completely ignore those that make money on our hard work and want to keep it all to themselves, and my responsibility to not prioritize the needs of those that contributes over those that are selfish..

OSS is also about giving back, and this year it’s what is being enforced on the OpenEverything projects. Fostering contributions is what we want to get to. No more free-lunch though, that’s for sure.

Conclusion

This new organization should let OpenRasta absorb patches and get on the 2.1 release cycle faster, and then on to 2.2, etc. This will allow me to focus on what I want to build for 3.0, and this will also allow more contributors to own projects more directly than before.

If you have comments or feedback on all those new plans, please let us know on the now unified openeverything-dev mailing list.

Please don’t use it for user questions, we’ve moved all support of all those frameworks to stackoverflow where google can do its job and users can find their answers more quickly.

The details are at http://www.gep13.co.uk/blog/?p=609 so if you’re interested in HTTP or/and ReST, do come along!

As I highlighted before, if you’re anywhere in Europe where you know of a user groupb, I’ll be happy to come and share what I learnt in the last two years building composite systems and how to build them using a package manager. Just email me at seb at serialseb dot com, and let’s make it happen!

Before

One of the things that has been difficult in the past with OpenWrap was knowing exactly what dependencies you had and how versions got selected. Worse than that, while in a project, the list-wrap command would only show you the list of packages currently in your folder, be it that they are in use or not.

A better list-wrap

So we’ve worked very hard to give you a better way to see what’s going on from the command line (and it’s always my belief that if you can make something understandable in 80*120 characters, you can make it understandable in *anything*).

Listing project packages

The first change is when you issue a list-wrap command within a project.

C:\src\openwrap [exp_list_wrap +5 ~25 -4 !]> o list-wrap # OpenWrap Shell 2.0.0.10 # Copyright © naughtyProd Limited 2009-2011 # Using C:\src\openwrap\wraps\_cache\openwrap-2.0.0.86438072\bin-net35\OpenWrap.dll (2.0.0.1) ─default scope ├─SharpZipLib 0.86.0 ├─openfilesystem 1.0.0.61263243 ├─openwrap 2.0.0.86438072 ├─tdnet-framework 2.0.0.48555719 └─Mono.Cecil 0.9.4.1 ─tests scope ├─NUnit 2.5.9.10348 ├─SharpZipLib 0.86.0 ├─openfilesystem 1.0.0.61263243 ├─openwrap 2.0.0.86438072 ├─tdnet-framework 2.0.0.48555719 └─Mono.Cecil 0.9.4.1

A couple of things are visibly different 1.0.

First, we now show the currently-used package for each of the scopes. if you don’t know what scopes are, I’d refer you back to the wiki documentation, the feature is one of the first we built for the now defunct 1.1, and will be blogged about when I consider the feature *complete*.

Second, we show you the packages *currently in use*. No more showing you lots and lots of package versions that are available but not used anymore, we only show you *the same thing your projects will get*.

Third, we show it as a tree. I love ASCII art and thought it absolutely had its place in a package manager.

Gimme gimme gimme a packaaaa aaage per line…

Of course, that’s also a vast improvmenet on what we had before. But as you know, package resolution ca get a bit hairy when a lot of packages have different rules, and trying to check manually what the heck is going on requires checking the package itself, something I’m sure you don’t want to be doing.

So now we can look at the whole of the resolution tree by adding an –includeDependencies flag. What’s the output then, I hear you ask with excitement and awe in your voice?

C:\src\openwrap [exp_list_wrap +5 ~25 -4 !]> o list-wrap -iD
# OpenWrap Shell 2.0.0.10
# Copyright © naughtyProd Limited 2009-2011
# Using C:\src\openwrap\wraps\_cache\openwrap-2.0.0.86438072\bin-net35\OpenWrap.dll (2.0.0.1)

─default scope
 ├─depends: sharpziplib = 0.86
 │ └─SharpZipLib 0.86.0
 ├─depends: openfilesystem = 1.0
 │ └─openfilesystem 1.0.0.61263243
 │   └─depends: openwrap content
 │     └─openwrap 2.0.0.86438072
 │       ├─depends: sharpziplib = 0.86
 │       │ └─SharpZipLib 0.86.0
 │       ├─depends: openfilesystem = 1.0
 │       │ └─openfilesystem 1.0.0.61263243
 │       │   └─...
 │       ├─depends: openwrap content = 2.0
 │       │ └─openwrap 2.0.0.86438072
 │       │   └─...
 │       ├─depends: tdnet-framework
 │       │ └─tdnet-framework 2.0.0.48555719
 │       └─depends: Mono.Cecil = 0.9.4
 │         └─Mono.Cecil 0.9.4.1
 ├─depends: openwrap content = 2.0
 │ └─openwrap 2.0.0.86438072
 │   ├─depends: sharpziplib = 0.86
 │   │ └─SharpZipLib 0.86.0
 │   ├─depends: openfilesystem = 1.0
 │   │ └─openfilesystem 1.0.0.61263243
 │   │   └─depends: openwrap content
 │   │     └─openwrap 2.0.0.86438072
 │   │       └─...
 │   ├─depends: openwrap content = 2.0
 │   │ └─openwrap 2.0.0.86438072
 │   │   └─...
 │   ├─depends: tdnet-framework
 │   │ └─tdnet-framework 2.0.0.48555719
 │   └─depends: Mono.Cecil = 0.9.4
 │     └─Mono.Cecil 0.9.4.1
 ├─depends: tdnet-framework
 │ └─tdnet-framework 2.0.0.48555719
 └─depends: Mono.Cecil = 0.9.4
   └─Mono.Cecil 0.9.4.1

Now, that’s a lot of information, but you should be piping this stuff to less anyway. The –includeDependencies flag (which I’ve shortened to –iD here, thanks to our great console algorithm that does the right thing most of the time) lets you see *exactly* what happens.

From your original descriptor (“default scope”), you can see all the dependencies you have declared, as you’ve specified them. For each of the packages that were found in your current project for those dependencies, you see their dependencies too, and on and on. The only time we don’t show something is when the tree is recursive (say, openwrap depending on openwrap), in which case we show you the “…” line you can see above.

Remote packages

There’s a bit of changes to the remote packages too.

First, we’ve fixed the silly bugs where you got all the remote packages even if you specified –remote name, A very annoying stupid issue that appeared in 2.0 when we rewrote the remote support (yes, I do know that’s not been blogged about either yet, be patient, we’re nearly there).

Second, we’ve changed the displaying to show you what packages were found in which repository, so you can understand where things come from without cluttering the actual output. Say I have two remotes, remote1 and remote2, and they each have copies of openwrap. A simple o list-wrap –remote is now much easier to understand.

C:\src\openwrap [master]> o ls-wrap -query openwrap -rem
local
└─openwrap (1.0.2, 1.0.1, 1.0.0)
└─openwrap-bootstrap (1.0.0)

local
└─openwrap (2.0.1, 2.0.0, 1.1.0)

Couple of things have changed.

We now segregate packages by remote, so you can see immediately which remote has which package, making knowing where things come from much easier.

The second change is that we only show the first three components of package versions by default, as this is what OpenWrap understands and what you should rely on.

Conclusion

Understanding dependencies between packages can be a bit of a challenge. The new list-wrap should give you better understanding of what is actually going on.

If you’d like to see that workshop organized in your location, let me know where and we try and organize it. In the next couple of months, I’ll be in Sweden, Denmark, France, Belgium and Norway, so don’t hesitate to ask