Compiler Tokens

The C++ compiler analyses a program; it breaks the program into tokens. A token is a distinct unit that is recognisable by the compiler: There are five categories of tokens:

• Keywords 
• Literals 
• Identifiers
• Operators 
• Whitespace includes : blank , horizontal and vertical tabs , new line and form feed characters and comments 

Keywords are those words whose meaning has already been explained to the compiler Keywords should be considered to be reserved. Keywords are not allowed to be used as variable names 

There are 48 keywords for C++ language 

asm, auto, break, case, catch, char, class, const, continue, default, delete, do, double, else, enum, extern, float, for, friend, goto, if, inline, int, long, new, operator, private, protected, public, register, return, short, signed, sizeof, static, struct, switch, template, throw, try, typedef, union, unsigned, virtual, void, volatile, while

Identifiers :
• A valid identifier is a sequence of one or more letters, digits or underline symbols ( _ ).
• There is no limit to the length of an identifier, although for some compilers, only the first 32 characters of an identifier are significant.
• Neither spaces nor marked letters can be part of an identifier. Only letters, digits and underlined characters are valid.
• The underscore character '_' counts as a letter. In addition, variable identifiers should always begin with a letter. They can also begin with an underline character (_). In no case can they begin with a digit.
• case is significant, and all characters are considered significant.
• language's keywords should not be used to define an identifier.

Literals :

Literals are fixed values that appear directly in a program's code. They represent constant values of a particular data type.
Literal may be a character constant, integer constant, floating-point constant, or string constant.