So, I have been writing a bunch of posts over the last 14 months, how we have been using Agile, actually full Extreme Programming practices to build a multi-million dollar Enterprise Software platform and application for the banking sector. We certainly have seriously stressed Extreme Programming/Agile techniques to their limits as this is not a small piece of software, but a large Enterprise solution that gets sold into the top banks in the world. We certainly have proven that you can use Extreme Programming/Agile techniques to build a 1.8 million dollar Enterprise product family. have talked about being an Agile Architect and why it's neccessary, how we went to CTP in July, the Process we use, our tools, and even our failings.

So, after 48 Iterations we finished all the functionality we had agreed with Business was necessary for a "Phase I" delivery of our Next Generation/V5.0 product, as our Next Generation architecture will span an ambitious set of goals and products on top of this platform. Business and Development agreed together that we would stop and start a three week Iteration of fixing bugs in our backlog, testing and eating our dog food. In Extreme Programming, you are really not supposed to carry over bugs out of the Iteration but this was extremely hard with one week Iterations. We turned out very well overall as all the testing found just over 100 total bugs for 14 months work which is an order of magnitude less bugs than our previous product development techniques. In addition, we have over 1,000 unit tests and the code is well factored, clean and maintanable. The best part is the whole team understands it, not individuals.

I actually haven't written about it but I have been working as Agile Architect the last few months on the next phase and not as part of the Iterations directly. These involve a whole lot of Workflow, Reporting and much more. Anyhow, I made a stand with my management the last 3 weeks and insisted that I code and debug with the team. It is of course the right thing to do. So we are in ship mode. I have been using my 24 year experience with many other ship cycles to help us make sure we are doing everything neccessary to ship a first class product. Anyway, we are all in our War Room the last three weeks working hard and it always amazes me the level of teamwork we have and the will. No bullshit in the way. Business and Dev in the same room, fining issues, triaging on the spot and fixing them right away. No triage committees, no approval cycles, just teamwork.

We don't believe in Premature Optimization so we have used this 3 week cycle to work intensely on performance issues, especially me. Our technologies have not failed us. WCF has perfomed well under load and allows a ton of configuration and tweaking options - how many different freaking timeout settings are there?? -)). Wilson OR/M has been virtually perfect. I must admit, I am converted. 14 months ago, I was the traditional data driven Architect type of guy with only stored procs and a CRUD layer. I thought the OR/M would collapse under load. It certainly hasn't. In fact, the bottleneck right now is the client layer. We are running Oracle 10.2g on Red Hat Linux server in the back end, IIS 6/WCF and the Domain Layer on Win2K3 Enterprise and both of these boxes are about 15% CPU usage. The problem so far, is in our big CAB Smart Client (more on that in a future post) but we are seeing some 95% CPU spikes and some large GC Heap allocations (ObjectBuilder is an issue here) but we have been working on the performance by bring in the Asynchronous Web Services pattern and not having an area of the client block during a WCF call.

We have worked long hours these three weeks and we met the deadline last night. Its funny, but after all these years what would be the last roadblock? Yes, you guessed it. An extremely thorny COM Interop bug that was causing a crash in a related system. I just can never escape it. It will be the topic of the next blog post.

Technorati Tags:  Software Development, Software Architecture,  Agile, Agile Development, Extreme Programming, Windows Communication Foundation, WCF, OR/M, SOA, Microsoft