Table of contents
1.
Introduction
2.
What is Java StringTokenizer?
3.
Constructors of the StringTokenizer Class in Java
3.1.
Examples of Constructors of String Tokenizer in Java 
3.2.
2. Constructor with String and Custom Delimiter
4.
Methods of StringTokenizer Class in Java 
4.1.
 
4.2.
 
4.3.
Examples of Java String Tokenizer Methods
4.3.1.
1. hasMoreTokens()
4.3.2.
nextToken()
4.3.3.
3. countTokens()
5.
Implementation of StringTokenizer in Java
6.
Difference between the StringTokenizer and the split method
7.
Frequently Asked Questions
7.1.
Why should I use StringTokenizer?
7.2.
Is StringTokenizer thread-safe?
7.3.
Can StringTokenizer handle multiple delimiters at once?
7.4.
What happens to empty tokens in StringTokenizer?
8.
Conclusion
Last Updated: Oct 29, 2024
Easy

StringTokenizer in Java

Author Yashesvinee V
0 upvote
Career growth poll
Do you think IIT Guwahati certified course can help you in your career?

Introduction

String Manipulation is a common and important operation performed on Strings. Appending, splitting and replacing characters are some types of String manipulation operations. The StringTokenizer class is explicitly used to split strings into two or more tokens, with a delimiter as its reference. Delimiters are characters that divide a string into two or more parts. For example, a string that contains a comma can be separated into two parts with the comma as the delimiter. 

StringTokenizer in Java

Some common delimiters are comma ( , ), semi-colons ( ; ), slashes ( / \ ) and pipes ( | ). The whitespaces act as the delimiter unless specified. The StringTokenizer keeps track of the index of the last token internally and computes the next token based on this index.

What is Java StringTokenizer?

The behavior of an instance of the StringTokenizer class depends on a parameter known as the returnDelim flag of boolean data type.

  • When the flag is set to true, a maximal sequence of consecutive characters (token) split with respect to the given delimiter is returned. 
     
  • When the flag is set to false, the delimiter itself is considered a token. Thus, a sequence of characters or a delimiter would be returned as a token.


Class signature:

public class StringTokenizer
extends Object
implements Enumeration<Object>
You can also try this code with Online Java Compiler
Run Code

Constructors of the StringTokenizer Class in Java

There are three types of constructor in StringTokenizer Class in Java

  1. StringTokenizer(String str)
  2. StringTokenizer(String str, String delim)
  3. StringTokenizer(String str, String delim, boolean returnDelim)

Lets discuss them in detail:

  1. StringTokenizer(String str)
     
  • It constructs a string tokenizer with the default delimiter set for the given string.
     
  • The default delimiters are the space (  ), tab (\t), newline (\n), carriage-return (\r), and new-feed character (\f).
     
  • The delimiters are not counted as tokens.
     
public StringTokenizer(String str)
You can also try this code with Online Java Compiler
Run Code

 

2. StringTokenizer(String str, String delim)
 

  • It constructs a string tokenizer of the specified string with the characters present in the delim variable as the delimiters.
     
  • If delim is set as null, no exceptions are thrown during the creation of the string tokenizer. However, implementing other methods on the resulting string tokenizer will throw a NullPointerException.
     
public StringTokenizer(String str, String delim)
You can also try this code with Online Java Compiler
Run Code

 

3. StringTokenizer(String str, String delim, boolean returnDelim)
 

  • It constructs a string tokenizer with the characters of delim as the set of delimiters.
     
  • The returnDelim argument, as mentioned before, is a flag that determines if the delimiters would be considered tokens or not.
     
public StringTokenizer(String str, String delim, int returnDelim)
You can also try this code with Online Java Compiler
Run Code


Practice this code by yourself on Online Java Compiler.

Examples of Constructors of String Tokenizer in Java 

In Java, StringTokenizer is a utility class used to split strings into tokens. The class provides multiple constructors to customize how strings are tokenized. Here are examples of constructors of StringTokenizer with code implementations and their output:

  1. Constructor with Only String Input

This constructor tokenizes a string using default delimiters (space, tab, newline, etc.).

Syntax:

StringTokenizer(String str)


Code Implementation:

import java.util.StringTokenizer;

public class TokenizerExample1 {
    public static void main(String[] args) {
        String str = "Java String Tokenizer Example";
        StringTokenizer tokenizer = new StringTokenizer(str);

        System.out.println("Tokens:");
        while (tokenizer.hasMoreTokens()) {
            System.out.println(tokenizer.nextToken());
        }
    }
}
You can also try this code with Online Java Compiler
Run Code


Output:

Tokens:
Java
String
Tokenizer
Example

2. Constructor with String and Custom Delimiter

This constructor tokenizes a string based on a specified delimiter.

Syntax:

StringTokenizer(String str, String delimiter)


Code Implementation:

import java.util.StringTokenizer;

public class TokenizerExample2 {
    public static void main(String[] args) {
        String str = "Java,Python,C++,JavaScript";
        StringTokenizer tokenizer = new StringTokenizer(str, ",");

        System.out.println("Tokens:");
        while (tokenizer.hasMoreTokens()) {
            System.out.println(tokenizer.nextToken());
        }
    }
}
You can also try this code with Online Java Compiler
Run Code


Output:

Tokens:
Java
Python
C++
JavaScript

Methods of StringTokenizer Class in Java 

Methods

Type

Description

hasMoreTokens()booleanIt checks if any more tokens are available from the tokenizer’s string.
hasMoreElements()booleanIt returns the same value as the hasMoreTokens() method.
nextElement()ObjectIt returns the next token as part of the tokenizer in an object.
nextToken()StringIt returns the next token as part of the tokenizer in a string.
nextToken(String delim)StringIt returns the next token as part of the tokenizer in a string after switching to a new delimiter set.
countTokens()intIt returns the number of tokens remaining in the string. In other words, it counts the number of times the nextToken() method will be called before an exception is generated.

 

 

 

 

 

 

 

 

Examples of Java String Tokenizer Methods

The StringTokenizer class in Java provides methods to break a string into tokens. Here’s an overview of some commonly used StringTokenizer methods with code examples and outputs.

1. hasMoreTokens()

This method checks if there are more tokens available in the string.

Code Implementation:

import java.util.StringTokenizer;
public class HasMoreTokensExample {
   public static void main(String[] args) {
       String str = "Java String Tokenizer Example";
       StringTokenizer tokenizer = new StringTokenizer(str);
       System.out.println("Tokens:");
       while (tokenizer.hasMoreTokens()) {
           System.out.println(tokenizer.nextToken());
       }
   }
}
You can also try this code with Online Java Compiler
Run Code

 

Output

Tokens:
Java
String
Tokenizer
Example

nextToken()

This method returns the next token from the string.

Code Implementation:

import java.util.StringTokenizer;

public class NextTokenExample {
    public static void main(String[] args) {
        String str = "Java:Python:C++";
        StringTokenizer tokenizer = new StringTokenizer(str, ":");

        System.out.println("Tokens:");
        while (tokenizer.hasMoreTokens()) {
            System.out.println(tokenizer.nextToken());
        }
    }
}
You can also try this code with Online Java Compiler
Run Code


Output:

Tokens:
Java
Python
C++

3. countTokens()

This method returns the total number of tokens left in the StringTokenizer.

Code Implementation:

import java.util.StringTokenizer;

public class CountTokensExample {
    public static void main(String[] args) {
        String str = "Java Python C++ JavaScript";
        StringTokenizer tokenizer = new StringTokenizer(str);

        System.out.println("Total tokens: " + tokenizer.countTokens());

        System.out.println("Tokens:");
        while (tokenizer.hasMoreTokens()) {
            System.out.println(tokenizer.nextToken());
        }
    }
}
You can also try this code with Online Java Compiler
Run Code

Output:

Total tokens: 4
Tokens:
Java
Python
C++
JavaScript

 

You can also check about Java Tokens here.

Must Read Type Conversion in Java

Implementation of StringTokenizer in Java

import java.util.*;

public class Main
{
  public static void main(String args[])
  {
    String str = "The StringBuilder class in Java";
    System.out.println("String => " + str);

    System.out.println("\nConstructor1: ");

    StringTokenizer st1 = new StringTokenizer(str);

    System.out.println("Total number of tokens with space as the delimiter: " + st1.countTokens())

    while (st1.hasMoreTokens())
      System.out.println(st1.nextToken());

    System.out.println("\nConstructor2: ");

    StringTokenizer st2 = new StringTokenizer(str, " in ");
 
    while (st2.hasMoreTokens())
      System.out.println(st2.nextElement());

    System.out.println("\nConstructor3: ");

    StringTokenizer st3 = new StringTokenizer(str, " in ", true);

    while (st3.hasMoreElements())
      System.out.println(st3.nextToken("in"));
  }
}
You can also try this code with Online Java Compiler
Run Code

 

Output:

String => The StringBuilder class in Java

Constructor1: 
Total number of tokens with space as the delimiter: 5
The
StringBuilder
class
in
Java

Constructor2: 
The Str
gBu
lder class 
Java

Constructor3: 
The Str
i
n
gBu
i
lder class 
i
n
 Java

 

You can also read about the topic of Java Destructor

Must Read Conditional Statements in Java

Difference between the StringTokenizer and the split method

  • StringTokenizer is a legacy class that breaks strings into two or more tokens, while the split method splits a string according to the matches of regular expressions.
  • The StringTokenizer returns one substring at a time, while the split method returns an array of separated character sequences.
  • StringTokenizer, as a class, uses constructors to specify the delimiting character for a string.

Frequently Asked Questions

Why should I use StringTokenizer?

StringTokenizer is useful for quickly breaking down a string into tokens based on delimiters without needing to implement complex parsing logic. It's simple and efficient for processing basic string splitting in Java.

Is StringTokenizer thread-safe?

Yes, StringTokenizer is thread-safe because its methods are synchronized. However, it's generally recommended to use newer classes from java.util.regex or String.split() for thread safety, flexibility, and performance.

Can StringTokenizer handle multiple delimiters at once?

StringTokenizer can handle multiple delimiters but treats each delimiter character independently. It cannot interpret multiple-character delimiters or distinguish between delimiter sequences without additional logic.

What happens to empty tokens in StringTokenizer?

StringTokenizer ignores empty tokens between delimiters, so sequences like "a,,b" will tokenize as ["a", "b"]. If empty tokens are needed, consider using String.split() or java.util.Scanner.

Conclusion

The use of StringTokenizer class in recent times is not encouraged as they are not very flexible and robust as the split method of the java.util.regex class. However, it is still in use due to a few compatibility reasons and its execution speed. This blog explains the StringTokenizer class in Java. It also briefly discusses its constructors and methods along with their implementation.
Check out this problem - Longest String Chain

Related Articles

Strings in Java

Swap Function in Java

Hashcode Method in Java

Subsequence and Substring

String operations in Java

Solid Principles in java

Live masterclass