function defaultTokenizer(chars, wordOptions) {
            var word = chars.join(""), result, tokenRanges = [];

            function createTokenRange(start, end, isWord) {
                tokenRanges.push( { start: start, end: end, isWord: isWord } );
            }

            // Match words and mark characters
            var lastWordEnd = 0, wordStart, wordEnd;
            while ( (result = wordOptions.wordRegex.exec(word)) ) {
                wordStart = result.index;
                wordEnd = wordStart + result[0].length;

                // Create token for non-word characters preceding this word
                if (wordStart > lastWordEnd) {
                    createTokenRange(lastWordEnd, wordStart, false);
                }

                // Get trailing space characters for word
                if (wordOptions.includeTrailingSpace) {
                    while ( nonLineBreakWhiteSpaceRegex.test(chars[wordEnd]) ) {
                        ++wordEnd;
                    }
                }
                createTokenRange(wordStart, wordEnd, true);
                lastWordEnd = wordEnd;
            }

            // Create token for trailing non-word characters, if any exist
            if (lastWordEnd < chars.length) {
                createTokenRange(lastWordEnd, chars.length, false);
            }

            return tokenRanges;
        }
Problem: handling of trailing spaces before line breaks is handled inconsistently between browsers.

First, a <br>: this is relatively simple. For the following HTML:

1 <br>2

- IE and WebKit render the space, include it in the selection (i.e. when the content is selected and pasted into a
  textarea, the space is present) and allow the caret to be placed after it.
- Firefox does not acknowledge the space in the selection but it is possible to place the caret after it.
- Opera does not render the space but has two separate caret positions on either side of the space (left and right
  arrow keys show this) and includes the space in the selection.

The other case is the line break or breaks implied by block elements. For the following HTML:

<p>1 </p><p>2<p>

- WebKit does not acknowledge the space in any way
- Firefox, IE and Opera as per <br>

One more case is trailing spaces before line breaks in elements with white-space: pre-line. For the following HTML:

<p style="white-space: pre-line">1
2</p>

- Firefox and WebKit include the space in caret positions
- IE does not support pre-line up to and including version 9
- Opera ignores the space
- Trailing space only renders if there is a non-collapsed character in the line

Problem is whether Rangy should ever acknowledge the space and if so, when. Another problem is whether this can be
feature-tested