Craig Andera, over on PluralSight blogs (it has a main feed now!), has a post showing how he integrated Windows and Forms Authentication for the same web application. From his post:

“The trick was realizing that if you enable both “anonymous” and “integrated” authentication for a particular virtual directory, the browser won't try to authenticate to the web server until it receives a 401 (Unauthorized) back from the web server. But you can issue your own 401 any time you like!”

Well, the trick is not that you don’t get authenticated until a HTTP 401 error occurs, that’s standard IIS security. The trick is how he issues a 401 to force the authentication:

     if (user.Length == 0) // They haven't provided credentials yet
    { 
       Response.StatusCode = 401;
       Response.StatusDescription = "Unauthorized";
       Response.End(); 
     }

Check out the full code sample on Craig’s blog post on mixing forms and windows authentication.