Q: 3-state checkbox Win32 runtime When I add the BS_AUTO3STATE style to a default-styled checkbox on a dialog window like so at runtime;
this->Style |= BS_AUTO3STATE; // wrapper of Get/SetWindowLongPtr, ignore the specifics

.. it turns into a group box rather than a 3-state checkbox. What am I doing wrong?
Have I got the wrong control style?

A: This problem is caused by the fact that the BS_Xxx values are not actually defined in the headers to work as bit flags. Instead, their values just increase linearly:
#define BS_PUSHBUTTON       0x00000000L
#define BS_DEFPUSHBUTTON    0x00000001L
#define BS_CHECKBOX         0x00000002L
#define BS_AUTOCHECKBOX     0x00000003L
#define BS_RADIOBUTTON      0x00000004L
#define BS_3STATE           0x00000005L
#define BS_AUTO3STATE       0x00000006L
#define BS_GROUPBOX         0x00000007L
#define BS_USERBUTTON       0x00000008L
#define BS_AUTORADIOBUTTON  0x00000009L
// ... and so on

Note that BS_GROUPBOX (which is the style you're getting and don't want) is equal to 0x7. Your control is ending up with that style flag set because you're setting a combination of flags that works out to have a value of 0x7. Unfortunately, you can't just OR the flags together and get the result you desire.
Instead, you'll have to clear out the current button style using the BS_TYPEMASK flag and then set the individual BS_Xxx flag that you desire. For a normal checkbox, that is probably BS_AUTOCHECKBOX; for a 3-state checkbox, that is BS_AUTO3STATE.
Working sample code:
void ToggleCheckboxCtrl(HWND hwndCheckBox)
{
    // Retrieve the control's current styles.
    LONG_PTR styles = GetWindowLongPtr(hwndCheckBox, GWL_STYLE);

    // Remove any button styles that may be set so they don't interfere
    // (but maintain any general window styles that are also set).
    styles &= ~BS_TYPEMASK;

    // Just for example purposes, we're maintain our last state as a static var.
    // In the real code, you probably have a better way of determining this!
    static bool isRegularCheckBox = true;
    if (isRegularCheckBox)
    {
        // If we're a regular checkbox, toggle us to a 3-state checkbox.
        styles |= BS_AUTO3STATE;
    }
    else
    {
        // Otherwise, we want to go back to being a regular checkbox.
        styles |= BS_AUTOCHECKBOX;
    }
    isSet = !isSet;

    // Update the control's styles.
    // (You'll also need to force a repaint to see your changes.)
    SetWindowLongPtr(hwndCheckBox, GWL_STYLE, styles);
}

The Spy++ utility (bundled with Visual Studio) is an indispensable little utility for figuring out what is going wrong when toggling window styles. Run your app, and use Spy++ to locate the window and enumerate its current styles. Then change the styles, dump the new styles with Spy++, and see what went wrong.
