function data_state(c) {
            switch(c) {
            case 0x0026: //  AMPERSAND
                tokenizer = character_reference_in_data_state;
                break;
            case 0x003C: //  LESS-THAN SIGN
                if (emitSimpleTag()) // Shortcut for <p>, <dl>, </div> etc.
                    break;
                tokenizer = tag_open_state;
                break;
            case 0x0000: //  NULL
                // Usually null characters emitted by the tokenizer will be
                // ignored by the tree builder, but sometimes they'll be
                // converted to \uFFFD.  I don't want to have the search every
                // string emitted to replace NULs, so I'll set a flag
                // if I've emitted a NUL.
                push(textrun,c);
                textIncludesNUL = true;
                break;
            case EOF:
                emitEOF();
                break;
            default:
                // Instead of just pushing a single character and then
                // coming back to the very same place, lookahead and
                // emit everything we can at once.
                emitCharsWhile(DATATEXT) || push(textrun, c);
                break;
            }
        }
This file was partially mechanically generated from
http://www.whatwg.org/specs/web-apps/current-work/multipage/tokenization.html

After mechanical conversion, it was further converted from
prose to JS by hand, but the intent is that it is a very
faithful rendering of the HTML tokenization spec in
JavaScript.

It is not a goal of this tokenizer to detect or report
parse errors.

XXX The tokenizer is supposed to work with straight UTF32
codepoints. But I don't think it has any dependencies on
any character outside of the BMP so I think it is safe to
pass it UTF16 characters. I don't think it will ever change
state in the middle of a surrogate pair.