Cron and AppEngine

Quick PSA on using cron jobs with Google App Engine because it almost wreaked havoc for us.

App Engine has a lovely feature of having different versions of your app. You can upload a new version but not make it the default until you’re good and ready. We do this all the time for deployment. Deploy to a new version and try it out, then make it the default when we’re ready to unleash it. Often, we deploy to the new version a day or so in advance.

Cron jobs, it seems, are handled outside this versioning mechanism. If you upload a new cron.xml file, it’s active. Right now. Doesn’t matter if the version it was deployed in is the default or not. As soon as it’s uploaded, it’s the new cron-ness.

Where this almost bit us is that we added a new cron job in our most recent release (deployed yesterday but not active) to use a dynamic backend. As soon as the cron job got uploaded, it started running. I didn’t notice until this morning when our backend usage reflected the new cron job. Some quick research and here we are.

What this means long term is that cron.xml is no longer going to be deployed as part of our application anymore. It now becomes an entirely separate process. I’m a little annoyed that we have to wait until we pull the trigger on the new version before we can upload the new cron.xml but it’s a quick step.

Kyle the Mis-scheduled

Posted in Uncategorized | Tagged | 2 Comments

The Economics of Ergonomics

Let it not be said there are no downsides to living in the Bahamas (though if you’ll permit a little boasting, a shortage of fantastic venues if you’re lucky enough to be in a band is not one of them).

The desk where I work is too high, plain and simple. So much so that I’ve recently abandoned my Kinesis keyboard because it is not what you might consider “low form factor”. I’ve started feeling some twinges in my lower forearm that my unscientific diagnosis is attributing to the height of my hands while I type. Dumping the Kinesis has helped but it has also led to the return of stress in other areas of my hands. And no amount of raccoon skinning seems to alleviate the pain.

Getting a lower desk is easy enough but I’d actually like to do a little experimenting with two alternatives. Alas, neither are easily done in the Bahamas. The underlying problem is availability. The desks/equipment I want to test are not available here so I would have to order them in. Which means both shipping charges and import duties, the latter of which is a major source of income for the Bahamian government to offset the fact that there is no income tax. So returning said equipment is just not practical if it doesn’t work out. Nor is there much of a reseller market.

So I’m hoping I can get some comments from people who have done something similar.

Adjustable height desk

These are, of courses, desks where you can adjust the height easily. I like the idea of these for two reasons:

  • They can be set low
  • They can be set high

I’ve never tried a desk that you stand at but I’ve always wanted to. Working on my own, I tend to get up and wander a lot while I’m thinking. I also pace when I’m on the phone with someone for any length of time so it would be more convenient to walk up to the computer during the conversation should the need arise. (“You want to know the right pattern of plaid for a first date with your second cousin? Let me look that up.”)

I went desk-shopping over the weekend and the closest thing I saw was in an office supply store. And it wasn’t on the showroom floor. Off in the corner of the store were the employee desks. They were all essentially plywood based, laminated desktops all mounted in warehouse style shelving frames. They sat on brackets in the frame which means you could set the height to whatever you want. It wasn’t something you could easily adjust on the fly and my wife wasn’t too thrilled at the industrial look so it was a fleeting idea at best.

Command centrecomputer workstation furniture MYPCE Computer Workstation Furniture

This is an idea I’ve had ruminating in my head for a while now. I would get rid of the desk altogether in favour of a comfortable command-centre style or gaming chair. In front of it it, I’d mount my monitors on a couple of flexible arms somehow, possibly on the armrests or on stands on either side of the chair. The important thing is that I can slide the monitors out of my way when I want to get out of the chair, and slide them back in when I sit down.

The keyboard would rest either on my lap or on some flat surface on my lap. Or maybe go with a split keyboard (though one without a wire between the two) and have one piece mounted on each armrest. Haven’t quite worked out how the mouse would fit in though. A trackball on some little platform on the side makes sense but I’ve got one now and it doesn’t feel as productive as just a regular mouse.

I feel like this would be more comfortable and would reduce much of the muscle stress that seems to have become more prominent since hitting 40 earlier this year. All of this kind of makes sense in my head but the logistics of getting the stuff here is such that I don’t want to make the investment unless I’ve had a chance to try it out at least for a few days. There’s a chance my tendency to get up and wander might make this impractical. Or maybe cord management would be an ongoing problem.

The device shown at right, which I discovered while researching this article, is essentially what I’ve described. It’s some US$2750. Duty would add about 50% and shipping would likely bring the total price above five large. There’s another potential hurdle in that it may not be available anymore given the company’s domain seems to point to a parking spot. But even building my custom version will cost enough in non-refundable cash dollars for me not to head over eBay.

Instead, I hunt for a standard desk about four to six inches lower than the one I’ve got. Not as exciting, possibly not as ergonomic, but easier to replace.

So my question to you, my honorary hillbillies, for anecdotal evidence. Have you tried either of these devices? What’s good and bad? Good return for the money or does it sit in the garage next to the Bowflex you bought in a fit of New Year’s anxiety?

Kyle the Unreturnable

Posted in Sundry, Working Remotely | 24 Comments

Audit Fields in Google AppEngine

Executive summary: Here’s how we’re implementing audit fields in AppEngine. IT’S BETTER THAN THE WAY YOU’RE DOING IT!

I considered saying “I hope there’s a better way of doing it” but I believe I’ll get more responses if I frame it in the form of a challenge.

For all entities in our datastore, we want to store:

  • dateCreated
  • dateModified
  • dateDeleted
  • createdByUser
  • modifiedByUser
  • deletedByUser

Here are the options we’ve considered

Datastore callbacks/Lifecycle callbacks

AuditAppEngine supports datastore callbacks natively. If you use Objectify, they have lifecycle callbacks for @PrePersist and @PostLoad. The former works fantastic for dateCreated, dateModified, and dateDeleted. Objectify can handle all three easily as well provided you use soft deletes, which we do. (And they aren’t as bad as people would have you believe, especially in AppEngine. You’d be surprised how many user experience problems you discover strolling through deleted data.)

Both of these led to problems for us when we tried to use them for the createdByUser et al methods. We store the current user in the session and access it through a UserRetrievalService (which, at its core, just retrieves the current HttpSession via a Guice provider).

If we want to use this with the Objectify lifecycle callbacks, we would need to inject either our UserRetrievalService or a Provider<HttpSession> into our domain entities. This isn’t something I’m keen on doing so we didn’t pursue this too rigorously.

The datastore callbacks have an advantage in that they can be stored completely separately from the entities and the repositories. But we ran into two issues.

First, we couldn’t inject anything into them, either via constructor injection or static injection. It looks like there’s something funky about how they hook into the process that I don’t understand and my guess is that they are instantiated explicitly somewhere along the line. Regardless, it meant we couldn’t inject our UserRetrievalService or a Provider<HttpSession> into the class.

The next issue was automating the build. When I try to compile the project with a callback in it, the javac task complained about a missing datastorecallbacks.xml file. This file gets created when you build the project in Eclipse but something about how I was doing it via ant obviously wasn’t right. This also leads me to believe there’s something going on behind the scenes.

Neither of these problems is unsurmountable, I don’t think. There is obviously some way of accessing the current HttpSession somehow because Guice is doing it. And clearly you can compile the application when there’s a callback because Eclipse does it. All the same, both issues remaining unsolved by us, which is a shame because I kind of like this option.

Pass the User to Repository

This is what was suggested in the StackOverflow question I posed on the topic. We have repositories for most of our entities so instead of calling put( appointment ), we’d call put( appointment, userWhoPerformedTheAction ).

 

I don’t know that I like this solution (as indicated in my comments). To me, passing the current user into the DAO/Repository layer isn’t something the caller should have to worry about. But that’s because in my .NET/NHibernate/SQL Server experience, you can set things up so you don’t have to. Maybe it’s common practice in AppEngine because it’s still relatively new.

(Side note: This question illustrates a number of reasons why I don’t like asking questions on StackOverflow. I usually put a lot of effort into phrasing the question and people often still end up misunderstanding the goal I’m trying to achieve. Which is my fault more than theirs but still means I tend to shy away from SO as a result.)

Add a User property to each Entity

I can’t remember where I saw this suggestion. It’s kind of the opposite of the previous one. Each entity would have a User property (marked as @Transient) and when the object is loaded, this is set to the current user. Then in your repositories, it’s trivial to set the user who modified or deleted. This has the same issue I brought up with the last one in that the caller is responsible for setting the User object.

Also, when new objects are created, we’d need to set the property there as well. If you’re doing this on the client, you may have some issues there since you won’t have access to the HttpSession until you send it off to the server.

Do it yourself

This is our current implementation. In our repositories, we have a prePersist method that is called before the actual “save this to the datastore” method. Each individual repository can override this as necessary. The UserRetrievalService is injected in and we can use it to set the relevant audit fields before saving to the repository.

This works just fine for us and we’ve extended it to perform other domain-specific prePersist actions for certain entities. I’m not entirely happy with it though. Our repositories tend not to favour composition over inheritance and as such, it is easy to forget to make a call to super.prePersist somewhere along the way. Plus there’s the nagging feeling that it should be cleaner and more testable than this.

Related to this is the underlying problem we’re trying to solve: retrieve the user from the session. In AppEngine, the session is really just the datastore (and memcache) with a fancy HttpSession wrapper around it. So when you get the current user from the session, you’re really just getting it from the datastore anyway using a session ID that is passed back and forth from the client. So if we *really* wanted to roll our own here, we’d implement our own session management which would be more easily accessible from our repositories.

So if you’re an AppEngine user, now’s where you speak up and describe if you went with one of these options or something else. Because this is one of the few areas of our app that fall under the category of “It works but…” And I don’t think it should be.

Kyle the Pre-persistent

Posted in Google App Engine | Tagged | 8 Comments

The other side of AppEngine

I really do love Google AppEngine. That didn’t come out so well in a recent post. I’ve had a bad taste in my mouth ever since then and now that I’ve eliminated the dandelion wine as a potential culprit, I’m forced to conclude that I need to balance my hillbilly feng shui. So in no particular order, here is what I like about AppEngine:

Granular pricing

I complained about the confusing pricing in the last post but since then, a funny thing happened. BookedIN got some traffic (thanks in no small part to the Chrome Web Store.) We were getting feedback and adding features and fixing bugs and hey, howdy, hey, wouldja take a look at what’s happened to our AppEngine budget!

And for all the bitching I did on the different ways your application is charged, this jump in cost has led to some much-delayed but necessary shifts in how we code.

As an example, we are now hitting our datastore read quota regularly. Which leads to the question: “Why so many reads?” Which leads to a number of other questions most of which can be boiled down to “Why am I treating the datastore like a SQL database?”

Similar discussions have led to us taking much greater advantage of AppEngine features we’ve all but ignored, including task queues and the channel API.

This doesn’t mean we’re on a refactoring binge, mind you. Our costs have jumped to the point where we want to waste thousands of dollars in developer hours to save ourselves a few extra bucks every month. But it has affected our coding practices going forward. When discussing new features, there are new questions: Can we denormalize this object? Do we *really* need to send this email synchronously? Can we send a message back to the user via an open channel? That sort of thing. All of these questions we should have been asking but didn’t because we “didn’t have time”. Now that cash dollars are being spent and now that we’ve seen how little time it takes to tweak our way of thinking, we are now working towards a better performing and more scalable application.

Out-of-the-box services

I looked briefly at AWS. It’s very different from AppEngine. From what I can tell, it seems you get a server. What you do with it is up to you. With AppEngine, I have an environment for caching, task queues, logging, mapping, channels, etc.

The price you pay for all these services is that if you ever decide to move away from AppEngine, you’re rewriting. For the most part, this ain’t no generic framework you can swap in your own implementation fer.

Multiple simultaneous versions

You can deploy up to ten versions of an application to the same account. For us, that makes actual deployments very smooth (in theory). We deploy on Saturday nights because statistically, that’s when the app is least used. But any time the week prior, we can deploy to a new version in production and just not make it active. That allows us to do smoke tests (if possible) and make last minute tweaks (or, more likely, add new features). The night of, it’s mostly a matter of switching the default version from the old one to the new one. I say “mostly” because sometimes, we have mappers to run to transform the data in order to handle some new feature. If you structure your model properly, this is an optional feature, but I like to do it anyway. Even with our massive increase in traffic, this still takes less than 15 minutes.

There’s also now a new experimental feature that lets you redirect a percentage of your traffic to a different version. This is designed for A/B testing but something tells me it won’t be used as such. In any case, I haven’t tried it.

Backup and Restore

I’m including this one because my original post said it wasn’t there. Since writing it, an experimental backup/restore feature has been added to the Datastore Admin console. I’ve tried it and it works though the lack of blob support means that it’s usefulness for us is almost non-existent. I suspect it’s not far away from being ready though. In the meantime, there are other solutions. One is DatastoreBackup which looks promising. Another is Google App Engine Backup And Restore, which looks very out of date. I mention it only to serve as an example for a long-standing PSA I have: Acronyms are not always the best name for a software project.

(NOTE: I *still* have to use Firefox to access the Datastore Admin console because it usually doesn’t appear in Chrome.)


Final note: I lied about this list being in no particular order. For all my initial confusion, I can’t understate how much I now love the granular pricing of AppEngine. If we want to pay less for our hosting, there are any number of things we can do to restructure things. And the nice thing is: almost every one of them will result in a better performing application. Before this, I’ve always had a fuzzy feeling of “yes, I know we could optimize but how much is that *really* going to save us?” Now, it’s right there in front of me. “We’re using too many channel API calls.” That’s because there’s an error in our logic when we rejoin a channel. “Our front-end instance hours are going up.” Maybe stuff can be refactored into a task queue or a backend. “Our data reads have spiked.” Perhaps we should cache the list of countries instead of querying the datastore every time the user brings up a certain dialog.

Kyle the Discounted

Posted in Uncategorized | Tagged , | 5 Comments

QA: A Hillbilly Love Story

The hillbilly turned forty last weekend! I’ve been anticipating this giddily because now, I can be crotchety and people will still find me adorable.

I hate QA. Hate, hate, hate, HATE, HATE, HATE it. I despise it so much, I think I might swear. Here goes….I f—, okay, I can’t do it but I still &*%$ hate it.

Side note: I know all QA people weren’t raised by trolls and/or gnomes but I’m going to generalize here. If you are a QA person and were raised by evil leprechauns, please don’t take offense.

As a hillbilly, I’m an innately optimistic and forgiving person. An ugly codebase to me is a sea of opportunity. Changing requirements? Ha! I can put my OCD up against any client. I can see the positive in almost anything. Except the emotional rollercoaster that is QA.

The end of an iteration is a time of celebration and relief for me. We’ve completed a significant piece of functionality on our application. We’ve followed best practices, left the code in a better state than when we started, and all our unit and UI tests are passing (more or less). Despite my past experience, I always feel great pride when I email our QA department:

Hey there, QA folkery! I come bearing tidings of great joy! We have a new version of the application for your revelry and astonishment! You will surely faint in awe at the sheer glory of it! I await your token sign-off. Acclaim and praise is optional (but recommended).

After that, I sit quietly for a few minutes to bask in my accomplishment of the last couple of weeks then dive into the next iteration, the previous one long since gone from the vestiges of my memory by that point.

The inevitable spartan response:

Please fix the following issues:

  • As per the spec, emails should be sent every hour, not daily
  • I got a 404 error when I went to the public booking URL
  • I know this wasn’t part of the iteration but we should display a message when someone confirms an appointment. Can you include this?
  • You spelled it “cancelled” in the settings screen but “canceled” on the calendar.

Let me know when a new version is up for testing.

By now you will not be surprised when I tell you that the QA process was invented by the Marquis de Sade.

QA goblins: surely you see my point, no? The iteration is over. Done. Finished. There’s no “D.S. al fine” at the end. In my head, I’ve moved on to new functionality. I’ve got tests written and prototypes completed. And you want me to forcibly re-enter the past?!? All the hard problems from that iteration have already been solved. I don’t care about your “cosmetics” or your “functionality”. That new feature you’ve requested? It’s KILLER! It’ll be a great asset to the app. When I get around to it. That 404 error? Just a configuration issue. It’ll be fixed when we deploy. And come on, spelling mistakes? Have you not been on Facebook lately?

In short: I. Don’t. Want. To. Do. It. Right. Now.

And I have to. Virtually everything brought up in a good QA process is valid from bugs to cosmetic issues to features that we need to include this iteration even though they weren’t in the list. Without them, we don’t have a usable product. Users will complain or worse, move on to something else. But my headspace is somewhere else. I’ve already compartmentalized the new features and have shifted my own personal Eye of Sauron on them. Now, there’s leakage that I thought I’d never have to deal with again.

Another psychological issue: I’ve poured everything I’ve got to deliver the best that I’ve got and it wasn’t good enough. It’s never good enough. I’ve just presented to you the end result of four years of university training and over a dozen years of experience as an “expert in the field”. And with a few phrases, you send me slinking back to my computer to do it over again.

It’s a little strange in some regard. I welcome criticism of my coding choices and style. I actively seek it out sometimes. Even code reviews can be made fun again. But something about the subtly adversarial nature of QA raises my dander. The fact that it’s so essential to quality and that I can’t think of an effective way around it doesn’t help…

So to sum up:

  • QA forces me to change my headspace. And I hate it for that.
  • QA points out my flaws as a developer. And I hate it for that.
  • QA is necessary and makes software better. And I hate it most of all for that.

To end on a more positive note, most QA people (including those that do it as part of other duties) I’ve worked with are extremely nice. There’s no accusatory tone in the responses, no blame on messing things up, and no sense of “you’ve *really* screwed things up this time.” In their minds, they’ve done what’s expected and like me, have made the product better for it.

But I still hate you.

Kyle the Unadulterated

Posted in Uncategorized | Tagged , | 12 Comments