Generating a lexical analyzer using lex

A computer program often has an input stream of characters that are easier to process as larger elements, such as tokens or names. A compiler is a common example of such a program: It reads a stream of characters forming a program, and converts this stream into a sequence of items (for example, identifiers and operators) for parsing. In a compiler, the procedures that do this are collectively called the lexical analyzer or scanner.

Expressing the scanning task in a general-purpose procedural programming language is usually difficult. The scanning transformations are usually easy enough to describe; however, it is hard to express them concisely in these languages.