Code360 powered by Coding Ninjas X Naukri.com. Code360 powered by Coding Ninjas X Naukri.com
Table of contents
1.
Introduction 
2.
1. Keywords
2.1.
C++
3.
2. Identifiers
4.
3. Constant
5.
4. Strings
6.
5. Special symbols
7.
6. Operators
8.
Frequently Asked Questions
8.1.
What is a token used for?
8.2.
What are tokens in the compiler?
8.3.
What is the purpose of tokens in C++?
8.4.
What is the token function in C++?
9.
Conclusion
Last Updated: Jul 9, 2024
Easy

Tokens In C++

Author Geetika Dua
0 upvote
Leveraging ChatGPT - GenAI as a Microsoft Data Expert
Speaker
Prerita Agarwal
Data Specialist @
23 Jul, 2024 @ 01:30 PM

Introduction 

Understanding Tokens In C/C++

Keywords come in the category of the smallest elements of a program that are meaningful to the compiler. These elements are called tokens.

Also, see Literals in C.Fibonacci Series in C++

Tokens in C++ are the smallest units of a program that have meaning to the compiler. They include identifiers, keywords, constants, strings, operators, and punctuation symbols. The C++ compiler uses tokens to understand and process the source code during compilation, parsing it into meaningful components for further analysis and execution. Each token serves a specific purpose in the program's syntax and semantics, contributing to its overall structure and functionality.

There are 6  types of tokens in C++ :

  1. Keyword
  2. Identifiers
  3. Constants
  4. Strings
  5. Special symbols
  6. Operators

1. Keywords

Keywords in programming languages, including C and C++, are reserved words with predefined meanings and cannot be used as identifiers (variable names, function names, etc.). These keywords are essential to the language syntax and define control structures, data types, and other language constructs.

Look at the simple C++ code given below to add two numbers.

  • C++

C++

int main() {
int x, y, sum;

//taking the value of the two numbers
cout << "Enter the two integers you want to add: ";
cin >> x >> y;

// storing the sum of two integers in sum
sum = x + y;

// prints sum
cout << x << " + " << y << " = " << sum;

return 0;
}


Output:

Enter the two integers you want to add: 3
2
3 + 2 = 9

 

If we observe the code, we can identify certain words that are conventionally used in our codes very frequently. The word <int> and <return> are two such words. These are identified as keywords in C/C++. Keywords are predefined reserved words that have a special meaning to the compiler. These cannot be used as identifiers. 

Some of the reserved keywords in C/C++ are given below.

autodecltypeenumnot_eq bnamespaceoperatorbreakcase
autobitand bbitor bboolbreakcasecatchchar8_t c
char16_tchar32_tcompl bconcept cconstconst_castconsteval cconstexpr
static_caststructswitchtemplatethisthread_localthrowtrue
trytypedeftypeidtypenameunionunsignedxor bxor_eq b
returndynamic_castexplicitnullptrnoexceptstatic_assertintchar
booldeleteexport cor bnot bstaticprivatepublic
protecteddoexternor_eq bnot_eq bsizeoffalsetrue
ifdefaultforprivatevirtualsignedorelse
floatdoublefriendregister reinterpret_castvoidreturnwhilenew
enuminlinegotorequires cvolatileshortwchar_twhile
constinit ccontinueco_await cco_return cco_yield cusing declarationusing directiveasm a
Get the tech career you deserve, faster!
Connect with our expert counsellors to understand how to hack your way to success
User rating 4.7/5
1:1 doubt support
95% placement record
Akash Pal
Senior Software Engineer
326% Hike After Job Bootcamp
Himanshu Gusain
Programmer Analyst
32 LPA After Job Bootcamp
After Job
Bootcamp

2. Identifiers

Identifiers are symbols or words that one supplies to the variables, functions, types, classes, objects and other such components of one’s code. If we look again at the program to add two numbers in C++, we observe that to identify the value of the first number, we use the identifier ‘x’, for the second number, ‘y’ and for the sum of the two, we use ‘sum’. 

There are some rules that must be followed while using identifiers as tokens in C/C++ these are as follows:

  • Keywords cannot be used as identifiers. However, identifiers that contain a keyword are legal. For example, ‘Tint’ is a legal identifier, but ‘int’ is not. 
     
  • Identifiers are case-sensitive. Thus ‘FileName’ will correspond to a different memory address than ‘fileName’. 
     
  • The first character of an identifier must be an alphabetic character, either uppercase or lowercase, or an underscore ( _ ). Therefore, ‘2numbers’ is an illegal identifier. 
     

Each identifier has a scope or visibility. This scope is the region of the program in which this identifier can be accessed. It may be limited (in order of increasing restrictiveness) to the file, function, block, or function prototype in which it appears. 

Also see, Abstract Data Types in C++

3. Constant

A constant is a token in C that corresponds to a number, character, or character string that can be used as a value in a program. Every constant has a type and a value on the basis of which, constants are categorised into the following types:

  • Floating Point ConstantsIt is a decimal number that represents a signed real number. The representation of a signed real number includes an integer portion, a fractional portion, and an exponent. 
     
  • Integer Constants: It is a decimal (base 10), octal (base 8), or hexadecimal (base 16) number that represents an integral value. We use these to represent integer values that cannot be changed.
     
  • Character Constants: A “character constant” is formed by enclosing a single character from the representable character set within single quotation marks (‘ ‘). 
     
  • Enumeration Constants: The named integer identifiers that are defined by enumeration types are called enumeration constants. To read more on enumeration, you might want to refer to C enumeration declarations
     
//floating point constants
15.75 1.575E1	// = 15.75
1575e-2	                // = 15.75
2.5e-3	                // = -0.0025
25E-4	                // = 0.0025

//integer constants
28 0x1C	  // = Hexadecimal representation for decimal 28
034	          // = Octal representation for decimal 28

//character constants
char schar = 'x';	        // A character constant
wchar_t wchar = L 'x';	// A wide-character constant for the same character

4. Strings

In the C and C++ programming languages, a string token represents a series of characters enclosed within double quotes (" "). It is a basic data type used to represent textual information. String tokens receive their interpretation as character arrays, and their conclusion involves appending a null character ('\0') to signify the string's ending. It enables the handling, storage, and manipulation of textual data within a program. C and C++ offer a variety of functions and operations for manipulating strings, including tasks like combining, comparing, and extracting strings sections.   

//Initialize a character array with individual characters
char string1[50] = { 'c', 'o', 'd', 'i', 'n', 'g', 'n', 'i', 'n', 'j', 'a', 's' };

//Initialize a character array
char string2[12] = { "codingninjas" };

5. Special symbols

In the C and C++ programming languages, special symbols refer to specific characters with predefined meanings within the language syntax. These symbols have distinct purposes and are used to denote operations, control structures, and language constructs. Below are several frequently employed special symbols in C and C++:

Asterisk (*): used to create a pointer variable and multiply the two variables.

Colon (:): used to initialize a list.

Pre-processor (#): serves as a macro processor that the compiler uses to modify your program before the start of actual compilation.

Comma(,): separates multiple statements, such as individual parameters within function calls.

Bracket ([ ]): Opening and closing brackets are used in arrays to indicate one-dimensional and multi-dimensional arrays.

Braces ({ }): Indicates a code's starting and ending point in which multiple executable commands are added.

6. Operators

In C++, operators are symbols that represent specific actions or computations to be performed on operands. These operands can be variables, constants, or expressions. Operators facilitate various operations such as arithmetic, logical, relational, assignment, bitwise, and conditional operations, among others. Understanding the different types of operators and how they function is fundamental to writing efficient and concise C++ code.

  1. Arithmetic Operators: These operators perform basic arithmetic operations such as addition (+), subtraction (-), multiplication (*), division (/), and modulus (%). They manipulate numeric operands to produce a result based on the operation specified.
  2. Relational Operators: Relational operators compare two operands and evaluate to either true or false based on the relationship between them. Examples include less than (<), greater than (>), equal to (==), not equal to (!=), less than or equal to (<=), and greater than or equal to (>=).
  3. Logical Operators: Logical operators perform logical operations on boolean operands. They include logical AND (&&), logical OR (||), and logical NOT (!). These operators are used to combine multiple conditions and evaluate their truth values.
  4. Assignment Operators: Assignment operators are used to assign values to variables. The basic assignment operator (=) assigns the value on the right-hand side to the variable on the left-hand side. Compound assignment operators such as +=, -=, *=, /=, and %= combine arithmetic operations with assignment.
  5. Bitwise Operators: Bitwise operators perform operations at the bit level. They include bitwise AND (&), bitwise OR (|), bitwise XOR (^), bitwise NOT (~), left shift (<<), and right shift (>>). These operators manipulate the individual bits of integer operands.
  6. Conditional Operator (Ternary Operator): The conditional operator (?:) is a ternary operator that evaluates a condition and returns one of two values based on whether the condition is true or false. It provides a concise way to express conditional expressions.

Frequently Asked Questions

What is a token used for?

A token in programming serves as a fundamental unit of syntax, representing individual elements like keywords, identifiers, and operators.

What are tokens in the compiler?

In a compiler, tokens are discrete units of language elements, like keywords or operators, identified during lexical analysis for further processing.

What is the purpose of tokens in C++?

Tokens in C++ serve as the smallest units of a program that have meaning to the compiler. They include identifiers, keywords, operators, constants, and punctuation symbols, enabling the compiler to understand and process the source code during compilation.

What is the token function in C++?

In C++, the token function is not a predefined function. Instead, tokens are fundamental components of the language's syntax and semantics. Each token represents a specific element of the program, such as variable names, keywords, operators, or literals, enabling the compiler to parse and analyze the source code accurately.

Conclusion

Every program has certain tokens which are the smallest elements that are meaningful to the compiler. In C/C++ we have keywords, identifiers, constants, literals and punctuators as tokens. In this article, we discussed each of these in detail, along with examples. 

We hope that this blog on tokens in c/c++ has helped you more about the concept.

Recommended Readings:

Live masterclass