Keyboard Jedi on Vista x64

This morning I was getting ready to record a screencast about ReSharper 4 EAP. To make it easier for people to follow along, I launched Roy Osherove’s excellent utility, Keyboard Jedi. Rather than the expected result, this friendly dialog box greeted me:

image

Living La Vida x64

My main development machine is running Vista x64. I’ve used KeyJedi frequently on my Vista x86 laptop and never had a problem. So I immediately suspected a 64-bit compatibility issue. KeyJedi is a .NET 2.0 application, which means that it will execute on the 64-bit CLR if available. (The 64-bit CLR was introduced with .NET 2.0. You don’t have to do anything special. If you install .NET 2.0 or higher on 64-bit Windows, the 64-bit CLR is installed automatically.) The 64-bit CLR uses a 64-bit JIT compiler, which produces x64 (or IA64)* machine code. This is why things like Paint.NET’s Gaussian Blur, which involves a lot of 64-bit arithmetic, run faster on 64-bit Windows. The MSIL is identical, but on 64-bit platforms, the JIT produces 64-bit optimized code. For example, a 64-bit ADD can be done in a single instruction on 64-bit hardware, but requires multiple instructions on 32-bit. (N.B. If you’re running 32-bit Windows on 64-bit hardware, there is no way to access the 64-bit capabilities of the chip as the OS thinks it’s running on a 32-bit chip.)

* x64 is the 64-bit enhanced, x86-compatible instruction set introduced by AMD. IA64 is used by Intel’s Itanium processors and is incompatible with x86. So you have to recompile the world to use IA64. Once Intel realized that not everyone on the planet was willing to recompile their programs, they introduced EMT64 for the Pentiums and Core chips. EMT64 is functionally identical to x64 from AMD.

WoW

So on Vista x64, KeyJedi is running on the 64-bit CLR. But that doesn’t explain why it fails. Most programs, like Paint.NET, just work. What is KeyJedi doing that is special and incompatible with 64-bit Windows? It is registering global system hooks to capture keyboard and mouse messages. Registering hooks and receiving messages involves Win32 calls, which are handled by Michael Kennedy’s Global System Hooks in .NET library. This library includes both managed (Kennedy.ManagedHooks.dll) and unmanaged (SystemHookCore.dll) code. SystemHookCore.dll makes 32-bit Win32 calls and receives 32-bit callbacks. You cannot make 32-bit Win32 calls directly to the 64-bit Windows kernel. You need a translation layer, which is the Windows-on-Windows or WoW layer.

The WoW layer marshals 32-bit Win32 calls to and from their 64-bit equivalent, translating data structures on the way in and out. This marvelous piece of magic is built into 64-bit Windows and allows most 32-bit programs to execute on 64-bit Windows without problem. The WoW layer sits below all 32-bit processes happily marshalling system calls back and forth. If you’re running in a 64-bit process, such as the 64-bit CLR host, there is no WoW layer beneath you. So you cannot load 32-bit DLL, such as SystemHookCore.dll. (This is also why 32-bit kernel-mode drivers don’t work on 64-bit Windows. There is no WoW layer in the kernel. It exists between user and kernel mode.)

Now we know the problem. The question is how to fix it. We could create a 64-bit version of SystemHookCore.dll, but that would involve a lot of spelunking and debugging of unmanaged C++ code. Not exactly how I want to spend my morning. The other option is to force KeyJedi to run on the 32-bit CLR even on 64-bit Windows. Then we would have the WoW layer beneath us and SystemHookCore.dll could merrily assume that it is running on 32-bit Windows while the WoW layer takes care of all the 32-bit to 64-bit Win32 marshalling. So how do we force a managed application to run on the 32-bit CLR…

Any CPU vs. x86

The easiest technique is to modify your project settings in Visual Studio. Just go to Project Properties… Build… Platform target and change it from Any CPU to x86.

image

Now you might think, "Wait a second. I’m compiling to MSIL, which is processor independent." Yes, you are, but when you select Any CPU (the default), your program will load on the 64-bit CLR if available. By selecting x86, you are forcing the program to run on the 32-bit CLR. Then just recompile your program and problem solved. Just one problem… Roy never released the source. (Yes, I’ve emailed Roy and asked him to do recompile with x86 as the target platform.)

No Source, No Problem

We don’t want to modify the application, just ask it to run on the 32-bit CLR. Turns out that the project settings above just twiddle the CORFLAGS inside the PE file header of the executable. The .NET Framework SDK ships with a program called corflags.exe for just this purpose:

image

You’ll notice that 32BIT has a value of 0, which means Any CPU. We want to twiddle this to 1. One little problem. The assembly is signed. Twiddling even a single bit will invalidate the signature and the CLR loader will prevent the application from loading. If the assembly weren’t signed, you could just execute:

corflags KeyJedi.exe /32BIT+

ILDASM to the Rescue

Time to bring out the big guns. We’re going to disassemble KeyJedi.exe into MSIL. Hold onto your hats…

ildasm /all KeyJedi.exe /out=KeyJedi.il

This produces three files:

KeyJedi.il

KeyJedi.res

Osherove.KeyJedi.MainForm.resources

Time to hack some MSIL… Open KeyJedi.il in your text editor of choice and search for ".corflags". Change the line to:

.corflags 0x0000000b    //  ILONLY 32BITREQUIRED

We can’t compile the code yet because we don’t have the private key that matches the public key embedded in the MSIL. Search for .publickey and delete it. (You could change it to your own public key generated with sn -k, but there’s no reason that we need to sign the assembly.) Now we can re-compile the MSIL using ilasm:

ilasm /res:KeyJedi.res KeyJedi.il /output:KeyJedi-x86.exe

If we execute KeyJedi-x86.exe, we get:

image

Success!!! KeyJedi is now running on Vista x64.

Postscript

I’m not going to redistribute the recompiled binary because KeyJedi is Roy’s baby and the fix is really straightforward for him to make. Look to his blog for an update. My main purpose was to help people better understand 64-bit compatibility issues and some tricks that 64-bit Windows does so that, in most cases, you aren’t forced to recompile the world to run the programs you’ve come to depend on.

About James Kovacs

James Kovacs is a Technical Evangelist for JetBrains. He is passionate in sharing his knowledge about OO, SOLID, TDD/BDD, testing, object-relational mapping, dependency injection, refactoring, continuous integration, and related techniques. He blogs on CodeBetter.com as well as his own blog, is a technical contributor for Pluralsight, writes articles for MSDN Magazine and CoDe Magazine, and is a frequent speaker at conferences and user groups. He is the creator of psake, a PowerShell-based build automation tool, intended to save developers from XML Hell. James is the Ruby Track Chair for DevTeach, one of Canada’s largest independent developer conferences. He received his Bachelors degree from the University of Toronto and his Masters degree from Harvard University.
This entry was posted in .NET, featured. Bookmark the permalink. Follow any comments here with the RSS feed for this post.
  • Maksymilian Majer

    Simple and brilliant solution. Thanks for this :)

  • http://www.michaelckennedy.net/ Michael Kennedy

    Hey James! Good job tracking that issue down. When I wrote that library back in 2004, 64-bit was this weird thing that happened on high-end servers. It’s certainly becoming more of an issue. Thanks for the detailed info for all the users of Keyboard Jedi and my hook library.

  • http://www.ring3circus.com Greg Jenkins

    Excellent article – sorted out my Qliner hotkeys in no time :)
    Thanks!

  • Tom Waltz

    Thanks for the great information James.

  • http://www.jameskovacs.com james.kovacs

    @Andrew – Michael Kennedy’s “Global System Hooks in .NET library” article (linked to above) explains why he needs the unmanaged C++ code. (cf. 2nd paragraph of “Building the Library”) I haven’t looked into it enough to know whether you can do it solely from managed code using just P/Invoke.

  • http://www.andrewvos.com AndrewVos

    Sorry, by “pure .net” I mean with the use of PInvoke :)

    A MOUSE_LL or keyboard_LL hook can be used in .net. Plenty of code samples online, if you want to have a look.

  • http://www.jameskovacs.com james.kovacs

    @Andrew – As far as I know, you can hook keyboard events within your process, but not globally from .NET. If you can hook the keyboard globally just from .NET, how do you do it? I might have missed that addition to the BCL as it has grown rather large in recent years. (I’m always willing to learn a new trick!)

  • http://www.andrewvos.com AndrewVos

    Ahh ok, so it’s using a low level keyboard hook then.
    Wonder why they used a native c++ hook library then, this can be done in pure .net.

  • http://www.jameskovacs.com james.kovacs

    @Andrew – KeyJedi registers a global Windows hook for keyboard events. The quote from MSDN that you mention is applicable if you’re registering a hook DLL in another process. After registering the hook, we have the user press a key combination, which causes 64-bit Windows to execute a callback, which is intercepted by the WoW layer (since it’s a 32-bit process), which is received by the 32-bit SystemHookCore.dll (Kennedy), which is relayed to the registered callback function in KeyJedi. To verify that KeyJedi can receive keystrokes destined for 64-bit apps, I launched Paint.NET, which ran in a 64-bit process (confirmed with sysinternals’ Process Explorer), and then typed while Paint.NET had focus. KeyJedi displayed the keystrokes as expected.

  • http://www.andrewvos.com AndrewVos

    Sorry, maybe I should elaborate…

    The library is compiled to 32 bit native, so:
    SetWindowsHookEx can be used to inject a DLL into another process. A 32-bit DLL cannot be injected into a 64-bit process, and a 64-bit DLL cannot be injected into a 32-bit process. If an application requires the use of hooks in other processes, a 32-bit application must call SetWindowsHookEx to inject a 32-bit DLL into 32-bit processes, and a 64-bit application must call SetWindowsHookEx to inject a 64-bit DLL into 64-bit processes. Also, the 32-bit and 64-bit DLLs must have different names.

    Maybe I’m missing something?

  • http://diditwith.net Dustin Campbell

    This is great James, thanks! These instructions got me up and running in time for my presentation.

  • http://www.andrewvos.com AndrewVos

    Hmmm. Problem is, the hook will only work for 32bit apps now, or is that just the CALLWNDPROC hook?