Why CRUD might be what they want, but may not be what they need

A brief history of CRUD

The Rolling Stones once sung:

“You can’t always get what you want, But if you try sometimes, well you just might find, You get what you need”

I think the same is true of CRUD. Our users seem to ask for it and we blindly agree to their wishes and build it; however, its regularly not what they need.

The File Store

When computers first arrived the early commercial use was to try and remove the need for paper files. Some of us may be old enough to remember that offices has filing departments with staff, where when we began a piece of work we went to get the records that the company had on that customer, or policy, or project. There were problems with this approach: file stores took up a lot of space, and the mangement of paper based records, particulary archival material is expensive; often, someone else had the file out, buried on their desk, and it was not always clear who – an early form of pessimistic concurrency control – which often stopped you responding to requests; files were only as useful as the discipline of maintaing them, and keeping them in order – searching was difficult within a file if it was not ordered and there was no idea of seaching the file store itself.

So it was natural when we began to think about ways in which computers could help in the workplace to think about electronic filing systems. We would record the data around the customer, policy, or project on an electronic form instead of a paper one and the computer could store it for us. We saved paper, we saved staff, we saved time, and we added new functionality such as the ability to search.

The UIs that we designed were optimized to enter data. After all there was a lot of data to enter just to catch up on. The essential questions for UX design became tab order or number of clicks – just how fast could we enter data. We wanted bulk data entry grids, and forms that mirrored our paper based forms, because that was what we needed to get out of the physical filing store and into our electronic store

Throughout the paper told us what to do and the process was built around the form. The form was passed from person to person. Stickers attached to the form, or boxes on the file defined the workflow. The paper-trail was the business process. Indeed, early analysis techniques simply followed the paper trail. Initial systems just followed the existing paper based process. Someone took the paper, the insurance application form, and type it in to the terminal. Then a batch run moved the record to the next step, so the next person in line could perform the same step that they had at the computer.

But no one really thought of this workflow as something the computer could handle. We just did handoffs the same way we did with paper. Indeed we sometimes used paper as the ‘token’ that we passed around for the workflow and told you that you needed to enter the system and do some work. Or, later we became more sophisticated, and used email to do the same thing.

And for this perspective – the electronic filing store – Create, Read, Update, and Delete (CRUD) was the paradigm we needed. It described the lifecycle of a paper file perfectly and with our goal to automate that physical file store it was a capable model – or so it seemed.

Brave New World

Users tend to create electronic documents today. There is less paper. Inputs come from phone, email, the consumer directly accessing a web-based interface. Removing the phsyical file store is no longer behind the business case to provide IT systems. We no longer want to remove the old filing room and its associated clerks. Of course we might want to move away from our new shadow IT world of documents on shared drives and Excel spreadsheets.

The business goals around automation have changed too. To obtain the savings that justify the cost of IT savings, we want to enable smaller numbers of people to carry out the tasks formally done by many through not just electronic filing but through business process automation.

But once we shift the goals away from data entry to process automation, the shape of our architecture changes too. Less of the value comes from data entry today. It comes from providing a key business differentiator that allows us to outperform the competition in service, control of costs, management of supply chain, enabling new business models through disintermediation etc.

CRUD – a model for removing the old file store is less useful as a paradigm when we come to think about analysing and architecting a response to this kind of business process.

Think about how we force users to interact under a CRUD paradigm. They have procedures written in a manual, a checklist for the process. They have to edit this page,then edit that page etc. The system consists of CRUD screens where the users do the data entry. People run their workflow using email. Business process is the preserve of a priesthood who maintain their position by understanding ‘how things are done here’.

The problem here is that the behavior is not captured in the system, it is captured in the manual at best, more likely in the oral transmission of process from one worker to another. The business understands that it has workflow and behavior – they defined process with copies of memos, forms etc. We have not removed those steps by automating them, we have just  given them electronic forms. Pejoratively we are little better than the shadow IT offering written in Access; why engage an IT expert at all for that, all you do is add cost.

Much of the issue here is sheer laziness on our part. When I see tweets and blog posts saying: most business apps are CRUD, I hear, “we don’t want to work with the business to show them what IT could do for them”. Part of the reason is because it requires little skill to model CRUD; modelling business process is harder – but that is where the value lies. Think of early software development approaches that began with modeling the data, and then identified processes to act on that data. Is it any wonder we built CRUD based systems if we started by thinking about data. Data Flow diagrams, Process diagrams. These were the cornerstones of how we thought about the design phase of the software lifecycle and we have failed to escape from it. However, for our users recording data itself is rarely the end goal any more, completion of a business process is the goal.

Now sure, in some cases we might still have CRUD, for some kinds of reference data, and I’m sure a lot of people will respond to this with straw men of CRUD only applications. But be careful. In a lot of cases the customer thinks about records management, and they don’t know you could help them with processing, business rules, workflow (or are frightened of what that might mean for their jobs). Anywhere where there is a process in someone’s head, a word document, or excel spreadsheet that says, when this happens, you need to go into this page and fill in these values on the form, you have behavior, or intent that could and should be captured.

Let’s take a simple example: changing the state of a purchase order. What we tend to be saying is: the workflow for this process has reached a certain point and so it can move into this state. We can immediately see that we must have a number of things for this to occur: the purchase order must have some data that we need to validate it; we can also see that a number of things may be predicated on this occurring: we need to send the order to fulfilment, we need to begin the billing process etc. So this act, completing a purchase order, has both a workflow or business process and rules. A CRUD based approach to design tends to encourage us to leave these outside the system, with the user, in some set of manual checks or switching between parts of the application. What we want to look at is: is this process something we could automate, could the workflow and rules live in the system?

Of course a traditional objection to putting workflow and rules ‘in the system’ is that they can change. This causes some folks to suggest that they ought to be left out.

Unfortunately in the typical business this means that a given named person Alice, Mary or Bob is the person that understands how to drive the process, and is mission critical to the success of the enterprise. If they get hit by a bus its all over, and they can pretty much ask for a raise whenever they like, because we can’t afford to lose them. Process diagrams often end up with a stick figure which imply that ‘Alice does her magic’. Often that person is uncommunicative and secretive about the process – to ensure that they remain essential to the business. When they are absent, things fall apart.

No business really wants that.

And the reality is that we have plenty of options we can use to make workflow and rules more configurable – though we need to look at the tradeoffs between the cost to change the app against using a rules engine or workflow engine carefully by understanding the amount of likely change.

Enter Tasks: What to you want to do today.

We need to drop the focus on CRUD and move to a focus on tasks: what is it that the user wants to accomplish, and how can we help them accomplish that?

Let’s think about the case of a family doctor. In an appointment they tend to take the patient’s history, diagnose the illness, and prescribe a course of treatment. Now, in the UK the paper based way to achieve this used to be the Lloyd George envelope. Simply replicating the functionality of the Lloyd George envelope using a CRUD approach would give us screens for entering consultation notes, prescriptions… But there is a lot more we could do to help. If we treat diagnosis as a task, then we ought to consider searching previous medical history to see if anything shows up related to the patient’s complaint that may be of concern. There may be guidelines for these symptoms that we should be following to check that we are not missing illnesses like meningitis. When we prescribe we can list common drugs for the illness, and look for incompatibility with drugs that the patient is currently being prescribed in a drugs database. That may lead to us offering further advice, for example if the patient is using the contraceptive pill and we prescribe them antibiotics.

Or perhaps consider an on line retailer where the customer can go in and change their account details. If the customer changes their address, we might want to prompt the customer for in-flight orders and ask where they want them to be delivered – the old address or the new address.

Ask yourself the question: am I thinking about the intent of what the user wishes to accomplish?

The grid is particularly dangerous ground in this regard. It is rare that you can capture intent through a grid. if you are using a grid you are probably not focused on a task but on CRUD

Greg Young has a great discussion on the Task-based UI. It is one of the building blocks in a journey towards understanding the why of CQRS

About Ian Cooper

Ian Cooper has over 18 years of experience delivering Microsoft platform solutions in government, healthcare, and finance. During that time he has worked for the DTi, Reuters, Sungard, Misys and Beazley delivering everything from bespoke enterpise solutions to 'shrink-wrapped' products to thousands of customers. Ian is a passionate exponent of the benefits of OO and Agile. He is test-infected and contagious. When he is not writing C# code he is also the and founder of the London .NET user group. http://www.dnug.org.uk
This entry was posted in Uncategorized. Bookmark the permalink. Follow any comments here with the RSS feed for this post.
  • vic

    nice post! reminds me of something I read time ago on Carlo Pescio’s blog:
    http://www.carlopescio.com/2005/07/grids-lists-and-like.html
    and also here
    http://www.carlopescio.com/2007/03/fighting-useless-computer.html

  • Pingback: friday links « A Programmer with Microsoft tools

  • http://twitter.com/galleman Glen B. Alleman

    Ian,

    What is the Domain where CRUD is seen as undesirable. Here in Nuclear
    Power, DoD weapons systems, and DOE facilities, drawings, policies and
    procedures, work instructions have strict governance rules around them
    for all the obvious reasons – people can either by acident or on
    purpose. We want the documents to flow the CRUD rules for good reason.

    Maybe in other domains this is not the case. Could you provide some information where you’ve found CRUD to not be helpful?

  • http://twitter.com/galleman Glen B. Alleman

    Ian,
    What is the Domain where CRUD is seen as undesirable. Here in Nuclear Power, DoD weapons systems, and DOE facilities, drawings, policies and procedures, work instructions have strict governance rules around them for all the obvious reasons – people can either by acident or on purpose. We want the documents to flow the CRUD rules for good reason.

    Maybe in other domains this is not the case. Could you provide some information where you’ve found CRUD to not be helpful?

  • Shez

    ‘With software you codify the new process and workers adapt to it’ – in most blue chips change management is required with software or without software, and most managers are better at managing change without software – its one less party (the software supplier) to deal with.

    Even if the code is well written, and workflow is abstracted out perfectly, your not going to be able to release a change to a blue chip within a month, even if your still talking to the company who wrote the software for you 5 years ago.  As Nigelren says, a good manager can change this process in a day if it is paper based, or the system did not try to rigidly encode business processes.

    In my (perhaps unlucky?) experience it is a definite anti-pattern to try and enforce a change to a business process through software if you don’t have the people-management skills in place to manage that change without software – I’ve seen any number of process enforcing systems ‘implemented’ that are never used because the users refuse to change their workflow.

  • Anonymous

    I think the better point of the article is not about CRUD but about not being stuck in CRUD only mode. People (average business person) have no clue what a computer can really do. They don’t know that a program can do this or that if you want it to. For them and their BS in Business, all they know is they want to store data and look at it. aka CRUD. Software Engineers know that is just plain right out boring, and pointless, it’s why they made sticky notes. We want to drill that data, figure ways to have it become self aware. We want our data store to have an AI. To take repetitive tasks and offer automation. When I sit in a meeting listening to the next CRUD they want me to build I start thinking how I can bend it to do this or that, and simplify everyone’s life.  A great example I ran into though with CRUD is our phone switch, it stores everything that makes the switch work. We needed a diagram to outline and show how the calls made there way through the switch. The computer has all the data it does the process a billion times a day. But it’s purely CRUD with a workflow, if the developers went outside the CRUD box and thought about it this would be a great addition to the program. Hit a button get a call path diagram for review. But they were so focused improving the CRUD they didn’t improve the program.  That is the point to be made, build the crud then take a setup back and go ok now how can I make this CRUD work for me.

    For example where I work they wanted a database to store HR information, basic stuff, nothing fancy. I asked well you have this phone data, wouldn’t it be great for us to automate the process to take their phone stats and calculate it into who much each employee costs you, and compare it against your income in. Then flag staff in question on cost to income ratio, and don’t worry the UI allows you to define the parameters you look at for costs, and what it compares to come up with costs, and it allows you to define employee fields like years of service, and what you define as profits.  They scratched their head acted like I was just going to waste time, and said just build the HR database. 6 months later in my “free time” I had built the other stuff and in turned we found about 6 people who were 50% lower than rest of staff. Management did some more digging and came to decisions about the staff. Now everything I did was just CRUD with workflow. It’s all you can do with data is CRUD, but you can program logic into CRUD to self process and notify. CRSPNUD (create, read, selfprocess, notify, update, delete), or CRAND (create, read, Automate, Notify,Delete) I like the 2nd one better. (ponders trademarking CRAND) lol

  • Keith Cromm

    Get with the program, man!  ”CRUD” is soooooo yester-decade(s)!

    Go contemporary with “iMAD”, the synonym acronym of CRUD.iMAD keeps in step with everything starting with “i” these days.
    As well, we’re driven ‘mad’ with the “crud” we still deal with in systems development.
    iMAD = Insert, Modify, Acquire, Delete

    ‘nuf said.

  • Nigelren

    I think the problem comes when the application is built to follow a certain process, the routing is embedded into the code rather than abstracted out.  Sometimes changing a simple manual process may involve a lot of application changes to ensure it reflects the changes.  The application needs to be modified, tested, rolled out… the cost and time makes it more epic than a simple – you pass this paper to them instead of them type manual change.

  • Jorgeleo

    The villain is not CRUD processing. The villain is exclusive-focus-on-CRUD-processing.
    We add value by automating process, eliminating redundancy, and providing better safeguards. CRUD is one of the tools, but CRUD is not the whole app… there are other tools like  workflows, visualizations, etc.

  • http://www.mariopareja.com Pareja Mario

    How do you justify that changing a business process is more expensive if software is involved?  With software, you codify the new process and workers adapt to it; without software, you rely solely on _all_ workers understanding and applying the new processes.

  • Shez

    Business processes often change faster than software can, and are often changed by people ‘on the ground’ rather than people up top.  

    Systemising business processes can be an anti pattern because of this – all your doing is making it harder and more expensive for a business to adapt over time, even if the ‘stakeholders’ have managed to correctly summarise an entire business’s processes in the first place.

  • Chris Hayes

    I think what he means by CRUD is form based systems.  There’s no way you can build a system without CRUD.  To build systems purely based off of a clients current forms system is, at best, a starting point.  Modeling business needs should outweigh raw computer interpretation of a clients paper based system but that’s not CRUD.  I like your article I just think it’s misleading to use the term CRUD for the poor simplistic approach of purely converting a clients current paper system into a forms based system.  That’s poor IT.

  • Dmitri

    Some of us may be old enough to remember that “its” sometimes may be “it’s”.
    I think you missed the point that CRUD is a common denominator for DB model, not for business requirements. Yes, some users extrapolate that to the entire system, but you need to have some reference point – and they are rightly justified to use DB functionality as that point.  Take it easy on their BA.

  • Anonymous

    Good points on this article and some of the comments. On software projects I’ve worked on, there is often a huge chunk missing in the “workflow” design of UIs, or even a base understanding by the programmers about how things are done.

    But finding that workflow can be difficult; especially if you ask two users, and they each do it a different way… and then need a manager who can sign off and say, “This is how it’s going to be from now on”, without raising unnecessary ire from the users.

    So most developers don’t bother trying; which is probably the worst way to handle the situation, and ends up writing software that is merely adequate instead of amazing. Sigh.

  • Roymorien

    I am at a bit of a loss
    here. To my mind, it is a simple fact of life that if you have a data store …
    let’s not be coy … if you have a table in a database, then you almost
    automatically require the ability to CRUD that table (My list of functionality
    is Add, Delete, Change, Search, Sort and List … and have a menu of reports
    and queries). If nothing else, this provides a useful capability to validate
    and verify the completeness of the data content of that table, and if
    appropriate ‘lookups’ and ‘cascades’ are available to connect to associated
    data then that provides a more complete picture of the data correctness.

     

    But this does not then
    place a barrier in your way for extending that functionality into other ‘work
    process’ processing. I am at a loss to understand why the author seems to think
    that there is CRUD or there is ‘work flow’ and ‘work processing task’, but not
    both.

     

    Yes, Let’s think about the case of a family doctor.
    What is stopping him from having a system where he can search the patient’s
    family medical history. It seems to me that there are two clear situations
    here, which can be combined and coordinated. Enter and maintain the patients’
    data (CRUD processing) AND search the medical history … even go out to the
    Internet to appropriate medical diagnoses sources and drug databases. 

    As a developer, I certainly
    would not avoid replicating the functionality of the Lloyd George envelope
    using a CRUD approach, but I certainly would avoid ‘simply’ doing that.
    Everything that the author suggested in that paragraph that the doctor could,
    or should, have available, would be on the table for discussion. Less than this
    would imply I am not providing the professional advice and support that my
    client, the doctor, needs. But I would still develop a very obvious CRUD
    processing capability, which I could do very quickly and easily using
    pre-developed ‘template’ programs, or even a CRUD processing code generator.

    So, I agree with the author’s
    generally gist, but I fail to see why the villain is CRUD processing.

  • Ian Cooper

    The problem is one of managing change. IT represents a threat, because people worry about their loss of control. This is nothing new – we had it when we replaced file stores and typing pools. People became used to that over time. I doubt many business would say: let’s not automate the file store, because it limits our ability to change the filing system.

    Organizations where there is no automated process are just as vulnerable, if not more so that ones where it is automated, to loss of key personnel. In addition, a lot of process change is just micro-optimization for efficiency. Automating usually brings greater benefits.

  • Ian Cooper

    Hi Marcel,

    I worked in clinical software in the UK some years ago. You don’t but can be sued for negligence if you get it wrong – however software is much better than doctors in catching issues like drug incompatibilities.

    Automating workflow does have some downsides for employees, in that it records your mistakes. However, it has some compliance upsides, we record these mistakes rather than trying to bury them.

    Making workflow easy to change is a separate post, I can’t respond here. a lot depends on how frequently process changes, whether you can distinguish between process that changes infrequently and process that changes frequently.  However, DSL approaches can really be your friend here. I think that whether those involve visual workbenches or not depends on the author. Udi has a great post on this recently, though provided you have some sort of text based flow, I don’t think visual designers automatically hurt.

  • Javier Paniza

    > this means that a given named person Alice, Mary or Bob is the person that understands how to > drive the process
    But Alice has intelligence and creativity that allows her to face unexpected problems and to improve the process in a daily basis. If you treat people as interchangable pieces you will lose a lot.

  • Nigelren

    I sort of understand what your saying, but isn’t this what workflow applications have been doing for quite some time.  Although what they are actually doing is breaking down the process into smaller CRUD steps.  But this is where I find it difficult – isn’t that what people do?!  All CRUD is a label, most peoples lives are simply taken up by creating things etc.
    For years people have been writing systems that look at what people want to accomplish – if you shop online, websites tend to ( annoyingly ) ask if you want X as you’ve already bought Y.  But it’s still – you’ve created an order ( CRUD ) do you want to order something else ( CRUD ).

  • http://www.mariopareja.com Pareja Mario

    Great post! I am in the process of re-thinking some old CRUDdy code – you just gave me a bit of inspiration to try out a few different approaches.

  • http://profiles.google.com/mdpopescu Marcel Popescu

    Somewhat off-topic, but I’m pretty sure that if you try to write a program that tries to replace any part of what a doctor does, you can go to jail, at least in my country :)

    Back on topic, good point – automating the workflow is of course much better than just CRUD. However, in my experience this is very hard to get right, and a disaster when you get it wrong (hard to modify). The example I read was the drunken resignation email – in the automated system, you’re instantly out of a job; in the manual system, your boss can intercept the paperwork and just throw it away, assuming that in the morning you came to your senses.

    Thanks for the pointer to the task-based UI article, I missed that one.

  • JackM

    Great article, and I agree with the sentiment, but there is a problem with this that I have run in to often, in ‘the real world’. Several times at different workplaces I have suggested (or implemented) the automation of a process that consists of a some instructions in Word , copy and pasting last months Excel document and clearing it out, filling in the blanks with this months data, all that horrible stuff. 

    What has happened? There has been resistance as “no one else knows how it works, and if the process changes who will update your automated process it? Worse still, no-one will know how to do it manually once the system has been running for a while, and small bugs may become showstoppers”. This seemed a fair criticism given the team, where everyone could do a VLOOKUP, but no-one could tell VBScript from Greek, let alone know what to do with c# source sitting in a SVN repo.If we step in to a workplace try to model the whole business process in a app with a task based ui, have we developers then baked ourselves a job for life, as any new use case of the system has to be added by us? Once ‘IT development’ turn down the request to create a ‘close account for customer X’ task, you bet the users will get happy with some Word instructions headed “TO DELETE AN ACCOUN T” which tells them to “go into each of customer X’s open items and click delete”.

    I can’t help thinking that the approach has risks unless the team who use the applications are empowered to change it and change it very rapidly. Maybe that means developers need to be part of the team that use the app, or the app has to have a scripting interface over the top of it (can of worms, letting the users tweak the code!), I don’t know.

  • James Tryand

    Cracking, and clear article.