  We introduce flat automata for automatic generation of tokenizers. Flat
automata are a simple representation of standard finite automata. Using the
flat representation, automata can be easily constructed, combined and printed.
  Due to the use of border functions, flat automata are more compact than
standard automata in the case where intervals of characters are attached to
transitions, and the standard algorithms on automata are simpler.
  We give the standard algorithms for tokenizer construction with automata,
namely construction using regular operations, determinization, and
minimization. We prove their correctness. The algorithms work with intervals of
characters, but are not more complicated than their counterparts on single
characters. It is easy to generate C++ code from the final deterministic
automaton. All procedures have been implemented in C++ and are publicly
available. The implementation has been used in applications and in teaching.
