Tokens In C++

2. Identifiers

Identifiers are symbols or words that one supplies to the variables, functions, types, classes, objects and other such components of one’s code. If we look again at the program to add two numbers in C++, we observe that to identify the value of the first number, we use the identifier ‘x’, for the second number, ‘y’ and for the sum of the two, we use ‘sum’. 

There are some rules that must be followed while using identifiers as tokens in C/C++ these are as follows:

• Keywords cannot be used as identifiers. However, identifiers that contain a keyword are legal. For example, ‘Tint’ is a legal identifier, but ‘int’ is not. 
   
• Identifiers are case-sensitive. Thus ‘FileName’ will correspond to a different memory address than ‘fileName’. 
   
• The first character of an identifier must be an alphabetic character, either uppercase or lowercase, or an underscore ( _ ). Therefore, ‘2numbers’ is an illegal identifier. 
   

Each identifier has a scope or visibility. This scope is the region of the program in which this identifier can be accessed. It may be limited (in order of increasing restrictiveness) to the file, function, block, or function prototype in which it appears. 

Also see, Abstract Data Types in C++

3. Constant

A constant is a token in C that corresponds to a number, character, or character string that can be used as a value in a program. Every constant has a type and a value on the basis of which, constants are categorised into the following types:

• Floating Point Constants: It is a decimal number that represents a signed real number. The representation of a signed real number includes an integer portion, a fractional portion, and an exponent. 
   
• Integer Constants: It is a decimal (base 10), octal (base 8), or hexadecimal (base 16) number that represents an integral value. We use these to represent integer values that cannot be changed.
   
• Character Constants: A “character constant” is formed by enclosing a single character from the representable character set within single quotation marks (‘ ‘). 
   
• Enumeration Constants: The named integer identifiers that are defined by enumeration types are called enumeration constants. To read more on enumeration, you might want to refer to C enumeration declarations. 
   

//floating point constants
15.75 1.575E1	// = 15.75
1575e-2	                // = 15.75
2.5e-3	                // = -0.0025
25E-4	                // = 0.0025

//integer constants
28 0x1C	  // = Hexadecimal representation for decimal 28
034	          // = Octal representation for decimal 28

//character constants
char schar = 'x';	        // A character constant
wchar_t wchar = L 'x';	// A wide-character constant for the same character

4. Strings

In the C and C++ programming languages, a string token represents a series of characters enclosed within double quotes (" "). It is a basic data type used to represent textual information. String tokens receive their interpretation as character arrays, and their conclusion involves appending a null character ('\0') to signify the string's ending. It enables the handling, storage, and manipulation of textual data within a program. C and C++ offer a variety of functions and operations for manipulating strings, including tasks like combining, comparing, and extracting strings sections.   

//Initialize a character array with individual characters
char string1[50] = { 'c', 'o', 'd', 'i', 'n', 'g', 'n', 'i', 'n', 'j', 'a', 's' };

//Initialize a character array
char string2[12] = { "codingninjas" };

5. Special symbols

In the C and C++ programming languages, special symbols refer to specific characters with predefined meanings within the language syntax. These symbols have distinct purposes and are used to denote operations, control structures, and language constructs. Below are several frequently employed special symbols in C and C++:

Asterisk (*): used to create a pointer variable and multiply the two variables.

Colon (:): used to initialize a list.

Pre-processor (#): serves as a macro processor that the compiler uses to modify your program before the start of actual compilation.

Comma(,): separates multiple statements, such as individual parameters within function calls.

Bracket ([ ]): Opening and closing brackets are used in arrays to indicate one-dimensional and multi-dimensional arrays.

Braces ({ }): Indicates a code's starting and ending point in which multiple executable commands are added.

6. Operators

In C++, operators are symbols that represent specific actions or computations to be performed on operands. These operands can be variables, constants, or expressions. Operators facilitate various operations such as arithmetic, logical, relational, assignment, bitwise, and conditional operations, among others. Understanding the different types of operators and how they function is fundamental to writing efficient and concise C++ code.

1. Arithmetic Operators: These operators perform basic arithmetic operations such as addition (+), subtraction (-), multiplication (*), division (/), and modulus (%). They manipulate numeric operands to produce a result based on the operation specified.
2. Relational Operators: Relational operators compare two operands and evaluate to either true or false based on the relationship between them. Examples include less than (<), greater than (>), equal to (==), not equal to (!=), less than or equal to (<=), and greater than or equal to (>=).
3. Logical Operators: Logical operators perform logical operations on boolean operands. They include logical AND (&&), logical OR (||), and logical NOT (!). These operators are used to combine multiple conditions and evaluate their truth values.
4. Assignment Operators: Assignment operators are used to assign values to variables. The basic assignment operator (=) assigns the value on the right-hand side to the variable on the left-hand side. Compound assignment operators such as +=, -=, *=, /=, and %= combine arithmetic operations with assignment.
5. Bitwise Operators: Bitwise operators perform operations at the bit level. They include bitwise AND (&), bitwise OR (|), bitwise XOR (^), bitwise NOT (~), left shift (<<), and right shift (>>). These operators manipulate the individual bits of integer operands.
6. Conditional Operator (Ternary Operator): The conditional operator (?:) is a ternary operator that evaluates a condition and returns one of two values based on whether the condition is true or false. It provides a concise way to express conditional expressions.

Frequently Asked Questions

What is a token used for?

A token in programming serves as a fundamental unit of syntax, representing individual elements like keywords, identifiers, and operators.

What are tokens in the compiler?

In a compiler, tokens are discrete units of language elements, like keywords or operators, identified during lexical analysis for further processing.

What is the purpose of tokens in C++?

Tokens in C++ serve as the smallest units of a program that have meaning to the compiler. They include identifiers, keywords, operators, constants, and punctuation symbols, enabling the compiler to understand and process the source code during compilation.

What is the token function in C++?

In C++, the token function is not a predefined function. Instead, tokens are fundamental components of the language's syntax and semantics. Each token represents a specific element of the program, such as variable names, keywords, operators, or literals, enabling the compiler to parse and analyze the source code accurately.

Conclusion

Every program has certain tokens which are the smallest elements that are meaningful to the compiler. In C/C++ we have keywords, identifiers, constants, literals and punctuators as tokens. In this article, we discussed each of these in detail, along with examples. 

We hope that this blog on tokens in c/c++ has helped you more about the concept.

Recommended Readings:

• Dynamic Binding in C++
• C++ Interview Questions