Eventual Consistency and Set Validation

I was just writing an email to the cqrs group http://groups.google.com/group/dddcqrs and figured it might be useful to put it up here as well as its a very common question I get.

The initial question was

Here’s the example I’m using:
A system that handles user registration for 2 million+ active users. These users should be able to login with their email address and password. They should also be able to change their associated email.

I have the following design: 

Entities:
User(Guid Id, string email, string hashedPassword)

Events:
UserRegistered(Guid Id, string email, string hashedPassword)
UserEmailAddressUpdated(Guid Id, string email)

Commands:
RegisterUser(Guid Id, string email, string hashedPassword)
UpdateEmailAddressForUser(Guid Id, string email)

ViewCaches:
RegisteredEmailAddresses(emailAddress) – Used for client side validation on email prior to sending a RegisterUser command

When processing a RegisterUser command, I need to validate that no other user has registered with that email. How can I do that without loading every user in the system? I could use a view cache like the client side, but then I would have business logic outside of my domain. Any suggestions?

 

This is a very common question. There were many responses with various suggestions. Mine follows.

 

I am just replying to the last one on the list after reading through.

To me the most important concepts have been completely missed in this
thread and they are a big part of why eventual consistency is so cool
(it makes you think about things).

*What is the business impact of having a failure*

This is the key question we need to ask and it will drive our solution
in how to handle this issue as we have many choices of varying degrees
of difficulty.

Most of the time the business impact of such a failure is low and the
probability of it happening is low. If we query the eventually
consistent store at the time of submission (either from client or from
server as this is a big part of how one-way commands work) then our
probability of receiving a duplication is directly calculable based on
the amount of eventual consistency. We can drive this probability down
by lowering our SLA very often this is enough.

We can detect asynchronously if we broke our invariant. Imagine an
eventhandler that inserts into a table with a constraint. If it gets
an exception, we broke the constraint (note this is not really the
“read model” but the same db can be used if convenient, it is
important to note the distinction as if we scale to have 5 read models
we don’t have 5 of these …).

What do we do if we break the constraint? We need to come back to that
business impact statement above. For most circumstances, just raising
an alert to an admin etc is enough, these things are very low
probability of happening and are often not worth the time/cost of
implementing automatic recovery. Just imagine 1 username create out of
1,000,000 fails this way. How long would it take to automate the
process of handling the situation? Consider discussions with domain
experts etc. 5 minutes of admin time once a year is much better ROI in
most of these situations than a week of developer time to automate.

Continuing along it has now been decided that this has large enough
impact that it should be automated. The said process that finds the
duplicates could either raise an event DuplicateUsernameDetected or
directly call a command ResolveDuplicateUsername (which involves more
discussion). It is important to note that in either of these cases we
are discussing the “What” not the “How” it would never issue a command
“DeleteUser” etc, how to handle these situations is core domain logic
and should be modeled within the domain. In the username example
perhaps ResolveDuplicateUsername marks the user as not being able to
login (and as a duplicate) and it sends an email to the user saying
“Hey we screwed up but its your lucky day! you get to create a new
username …”

But even after all of this if from a business perspective the impact
is too high we can still make things consistent. We could drop in a
service to the domain that deals with a consistent set. This would of
course be the last resort as its the most complicated of these
solutions and brings with it many limiting factors in terms of our
architecture.

Udi had a great example of this in his explanation of 1-way commands.
It was an ATM that would spit out money having only read your balance
from an eventually consistent read model. The reason this can work is
that from a business perspective the risk is low (and it is built into
the business model itself). You have a bank account with me, I know
your SS# and all of your information. For people who overdraw their
accounts I will recover atleast 90% of the money that has been
overdrawn. On top of that I charge a fee for each overdraw that
occurs. For these reasons the business impact of such a problem is
low.

To sum up I just want to reiterate that this is a *good* thing.
Eventual consistency is forcing us to learn more about our domain. It
is forcing us to ask questions that are otherwise often not asked.

Consistency is over-rated.

 

This entry was posted in Uncategorized. Bookmark the permalink. Follow any comments here with the RSS feed for this post.

12 Responses to Eventual Consistency and Set Validation

  1. Greg says:

    @bob

    also what is the lesser of two evils? by adding 2 seconds of eventual consistency we can pretty much assure availability. Would you rather have a bad user experience for people who attempt to game the system or be pretty much constantly available for normal users (enough 9s to the point that its not probable to be down for the lifetime of the system?)

  2. Greg says:

    @bob

    Are you suggesting the user has no role in this? Where is their personal responsibility in this? but….

    Let’s say its 2 seconds eventual consistency… How exactly would a user overdraw their account unless they are making duplicate cards? Yes I am perfectly willing to have a bad experience for users who try to game the system. No normal user will get into problems, it will seem perfectly consistent to them as it takes more than 2 seconds to withdraw money from an ATM

  3. Bob says:

    Are you serious with your ATM example? Basically you’re saying it’s ok to screw the user with overdraft fees because you architectured a system using eventual consistency. Eventual consistency is fine for FaceBook and Twitter but not for ATM bank transactions and definitely not when the customer has to suffer because of bad system design decisions.

  4. Greg says:

    @Remi I think there are a few ways of doing this but an easy way of doing this would be to pull back the user based on username and password (assuming you are also salting passwords with something other than username the probabilities of getting a failure would be astronomical). Also if you sent an email you would know the userid at the time of sending the email.

    Make sense?

    Greg

  5. Remi Despres-Smyth says:

    Seems to me the main problem with this solution is that once the first account’s been updated, you have two identical usernames which cannot be distinguished.

    > we could also mark him for the next time he tried to login to bring him to the change username screen the same as the email would have brought him to.

    How do you distinguish which of the two users is logging in, and which should be redirected to change their email address?

    Remi.

  6. Remi Despres-Smyth says:

    Seems to me the main problem with this solution is that once the first account’s been updated, you have two identical usernames which cannot be distinguished.

    > we could also mark him for the next time he tried to login to bring him to the change username screen the same as the email would have brought him to.

    How do you distinguish which of the two users is logging in, and which should be redirected to change their email address?

    Remi.

  7. Greg says:

    a wag yes a very poorly designed system could do that. I prefer to not consider what harm very poorly designed systems can possibly harm but prefer to instead focus on what a reasonable design leads to :) That sounds like a case of “Dr it hurts when I do this”

  8. a wag says:

    You would have a short period in which you could sign up under someone else’s already existing username, right? Then if the server set the auth cookie off the username, you could log in as the first user.

  9. >>note this is not really the “read model”

    good you cleared that up, thats one fatal design mistake I have to correct in my system asap where I do have a case like this

  10. Chris Nicola says:

    Very well put, there is huge value in actually understanding the problem and context of the problem when designing any solution. Seems like common sense, but it is surprisingly uncommon.

    This scenario, and it’s subsequent solution, is the exact sort of thing that turns a lot of businesses away from the CQRS model as they feel it is far too risky to design a data store in this way. Mind you, this is entirely driven by an irrational fear of uncertainty people get from ‘eventual consistency’ but it is still a reality that is difficult for developers encouraging CQRS to confront. In the end I suspect most lose the argument once someone starts to invoke their need for things to be ‘ACID’.

    The very suggestion that we can just submit an invalid username to the database and deal with it later would probably paralyze most BA’s and product managers.

  11. Greg says:

    we could also mark him for the next time he tried to login to bring him to the change username screen the same as the email would have brought him to.

  12. noobie says:

    Erm, forgive me if I miss something here,
    but since in the example the user’s email IS the username, and the user could have mistyped his email address (he could have written joe@somewhere.com instead of joe2@somewhere.com), thereby generating a duplicate email(username) address, how would you be able to send him an email as a correcting/compensating action in the first place?

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>