Josh Flanagan (very smart guy and good basketball player who should start updating his blog again) took me to task a bit for my last two posts. Yesterday I slipped and made a bit of a repeat post about one of those gnarly this calls this which calls that situations. It jogged loose a memory this morning about an equally horrible mess I created on a project several years ago (which according to karma was rewritten by Josh this year).
Phase 2 of the project called for my newly constructed system to integrate to some B2B infrastructure via a pair of MQSeries queues. I had assumed all along that I would use a Java Stored Procedure inside Oracle 8i to access the MQSeries code with JMS. The code to do this was trivial and took about 15 minutes to do. Easy money. Then somebody thought to try the code inside the Oracle database and found out really quick that it wouldn’t run inside that environment. Some later research suggested that it would work as long as we configured the database engine differently. Since the system was in production we decided not to make the configuration change.
The obvious answer is to just write a windows service that polls the MQSeries queue for incoming messages and relays them to the system. My problem was that we were all VB6 and PL/SQL developers without enough C++ experience, so nobody really had the skillset to write a custom service. Plus I had an unnatural fear of custom windows services from my support days. The next best, and probably easiest, approach would have been just to create a batch program and have it kicked off by our existing scheduling infrastructure. I shied away from this too because the infrastructure team was stretched way too thin to support us in the timeframe we had available and I really wanted to avoid interacting with this infrastructure. Again, because we were all VB6 coders, so we struggled with the MQSeries access (at the time it was several dozen Win32 calls for a single read). We had problems with the way the infrastructure guys deployed MQSeries. Because MQSeries wasn’t very VB friendly, we had to use a component that our central architects had written for standard access to MQSeries from middle tier code. That component had plenty of its own problems and could only function on the same box as the MQSeries queue. As I recall, the resulting architecture looked something like this:
- PL/SQL stored procedure running at intervals from the internal Oracle scheduler doing a,
- HTTP GET request to an ASP page hosted from the MQSeries machine that,
- Called a VB6 DLL that called into MQSeries, pulled out the XML messages, and pushed them right back to the very same database for processing
I know that there was another link (HTTP POST?) in the chain somewhere, but I don’t remember what it was. It mostly worked and stayed in production in some form or another for at least a couple years until we started using webMethods A2A for communication. You can probably make the correct guess that it was fragile and caused no small amount of production support problems.
I learned a couple of hard lessons from the experience.
- That whole “keep it simple, stupid” thing (work in progress)
- Watch out for technical risks. One of the things that scares me about XP/Scrum iteration planning is scheduling stories strictly by business value instead of technical risk. I think there’s a little room to negotiate scheduling for cases like this as long as the team, the PM, and the customer trust each other.
- Challenge your assumptions early. Earlier prototyping or spiking would have exposed or eliminated the problem.
- Be very cautious about new technology that you have not used before. Understand a technology first instead of just copying example code from a book somewhere.
- Either avoid technology that is completely new to your team or organization when there’s an existing alternative, or prepare to invest some time in familiarizing the team with the technology
- Don’t design what you can’t code.
- Don’t be afraid to ask for outside help. I was very uncomfortable in asking for external help on that project. Partly because I knew the organizational sluggishness wouldn’t give us the help we needed in a timely manner anyway, and partly because I was afraid the project would be taken away from me. In a large organization you have to do more looking ahead to identify where you’ll need help from shared functional teams (DBA’s, integration teams, etc.) because there always overbooked. You simply can’t just walk up to them and get something done right then and there. We also had a bit of an issue where a much larger, higher priority project was hogging all of the shared resources. Since they were sleeping at the office in cots in the training room, I can’t complain too much about that one.
- When your business model derives much of its profitability from an efficient supply chain, don’t hand a mission critical supply chain automation project to an inexperienced development team with a green technical lead (me).
Of course, after this was all over an Oracle DBA told me that the configuration changes to the database server would have been no big deal. If I’d been able to do it my original way the chain of components would have been much, much simpler.