Using hyperparameter tuning

Code Checking in your training application

Manage the hyperparameter command-line arguments in your application and submit your hyperparameter measure to AI Platform Training.

Handling the command-line arguments

When calling your training application, AI Platform Training sets command-line arguments. Put command-line arguments to use in your code:

1. Give each hyperparameter argument a name and parse it with your preferred argument parser. The names of the arguments must coincide with the parameter names you supplied in the task configuration, as previously mentioned.
2. Give the hyperparameters in your training code the values from the command-line arguments.

Reporting your hyperparameter metric to AI Platform Training

Whether you use TensorFlow for training or not will affect how you provide your hyperparameter measure to the AI Platform Training service. Additionally, it depends on whether you are training with a runtime version or a customized container.

A runtime version of TensorFlow 

You can submit your hyperparameter metric to AI Platform Training by publishing the metric to a TensorFlow summary if you train using TensorFlow and utilize an AI Platform Training runtime version. One of the following functions should be used:

• tf.compat.v1.summary.FileWriter.add_summary
• tf.summary.scalar

The following examples given below show the basics of two different ways to write your hyperparameter metric to summary. Both the instances assume you are training a regression model, and they write the root-mean-square-error between the ground-truth labels and evaluation predictions as a hyperparameter metric named metric1.

Keras

class MyMetricCallback(tf.keras.callbacks.Callback):

    def on_epoch_end(self, epoch, logs=None):
        tf.summary.scalar('metric1', logs['RootMeanSquaredError'], epoch)

logdir = "logs/scalars/" + datetime.now().strftime("%Y%m%d-%H%M%S")
file_writer = tf.summary.create_file_writer(logdir + "/metrics")
file_writer.set_as_default()

model = tf.keras.Sequential(
    tf.keras.layers.Dense(1, activation='linear', input_dim=784))
model.compile(
    optimizer='rmsprop',
    loss='mean_squared_error',
    metrics=['RootMeanSquaredError'])

model.fit(
    x_train,
    y_train,
    batch_size=64,
    epochs=10,
    steps_per_epoch=5,
    verbose=0,
    callbacks=[MyMetricCallback()])

Estimator 

# Create metric for the hyperparameter tuning
def my_metric(labels, predictions):
    # Note that the different types of estimator provide different different
    # keys on the predictions Tensor. predictions['predictions'] is for
    # regression output.
    pred_values = predictions['predictions']
    return {'metric1': tf.compat.v1.metrics.root_mean_squared_error(labels, pred_values)}

# estimator creation to train and evaluate
def train_and_evaluate(output_dir):

    estimator = tf.estimator.DNNLinearCombinedRegressor(...)

    estimator = tf.estimator.add_metrics(estimator, my_metric)

    train_spec = ...
    eval_spec = tf.estimator.EvalSpec(
        start_delay_secs = 60, # start evaluating after 60 seconds
        throttle_secs = 300,  # evaluate every 300 seconds
        ...)
    tf.estimator.train_and_evaluate(estimator, train_spec, eval_spec)

Hyperparameter tuning job details while running

You can keep track of hyperparameter adjustments by obtaining the detailed status of your ongoing training job.

A training job with hyperparameter tuning sets the following values for the TrainingOutput object in the response's Job resource:

isHyperparameterTuningJob is true by default.

A list of HyperparameterOutput objects is present in the field trials.

The TF_CONFIG environment variable is another place where you can find the trial ID. See the tutorial on obtaining information from TF CONFIG.

Getting Results

You may check the outcomes of each trial when the training runs are finished in the Google Cloud console. To obtain the results, you may use call projects.jobs.get. The metrics for all runs are contained in the TrainingOutput object in the task resource, with the metrics for the best-tuned run being identified.

The job description contains the findings of each study. You may filter trials by rmse, learning rate, and training steps via the Google Cloud console. Find the trial that produced the hyperparameter metric's ideal value. You can utilize the hyperparameter values displayed for that trial in subsequent runs of your model if the trial passes your criteria for model success.

The HyperparameterOutput FAILED trial status can either indicate that the training for that trial was unsuccessful or that trial failed to report the hyperparameter tuning measure. In the latter scenario, the parent job may still be successful even if the trial fails. You can check the trial log to see if the trial's training was unsuccessful.

Setting Limits to the number of trails.

The maxTrials value in the HyperparameterSpec object should be configured according to how many trials you want the service to be able to execute.

When determining how many trials to permit, there are two competing interests to take into account:

1. time (and consequently cost)
2. accuracy

You shouldn't set your maximum to a value lower than ten times the amount of hyperparameters you utilize to get the most out of the hyperparameter tuning.

Failed trials Handling

The training task may need to terminate early if your hyperparameter tweaking trials fail. You can specify the number of unsuccessful trials by setting the maxFailedTrials field in the HyperparameterSpec. AI Platform Training ends the training project if these many tries are unsuccessful. The Values must be less than or equal to maxTrials and maxFailedTrials.

AI Platform Training handles failing trials according to the following principles if maxFailedTrials is zero or not set at all:

• If the first attempt at your work fails, AI Platform Training terminates it immediately. Further trials will likely encounter failure because the initial trial's failure points to a potential issue with your training code. By terminating the work, you can identify the issue without needing to conduct additional tests or incur additional expenses.
• If the first trial succeeds, then AI Platform Training might end the job after failures during subsequent trials based on at least one of the following criteria:
  • The number of the failed trials has grown too high.
  • The ratio of the failed trials to successful trials has grown too high.

Running parallel trials

By specifying maxParallelTrials in the HyperparameterSpec object, you can determine how many trials will be performed concurrently.

Running parallel trials reduces the time the training job takes (real-time—the total processing time required is not typically changed). However, running in the parallel can reduce the effectiveness of tuning jobs overall. That is because hyperparameter tuning uses the results of the previous trials to inform the values to assign to the hyperparameters of the subsequent trials. When running in parallel, some trials starts without benefiting from the results of any trials still running.

Trails stopping

Set the enableTrialEarlyStopping value in the HyperparameterSpec = TRUE to allow early trial termination.

Resuming a completed hyperparameter tuning job

A finished hyperparameter tuning task can be continued to start from a partially optimized state. This enables the reuse of the expertise amassed during the prior hyperparameter tweaking.

To continue a hyperparameter tuning job, submit a new one with the setup shown below:

• Set the job ID of the prior trial as the value for resumePreviousJobId in the HyperparameterSpec.
• Specify values for maxTrials and maxParallelTrials.

The resumePreviousJobId configuration is used in the examples below:

gcloud

trainingInput:
  scaleTier: CUSTOM
  masterType: complex_model_m
  workerType: complex_model_m
  parameterServerType: large_model
  workerCount: 9
  parameterServerCount: 3
  hyperparameters:
    enableTrialEarlyStopping: TRUE
    maxTrials: 30
    maxParallelTrials: 1
    resumePreviousJobId: [PREVIOUS_JOB_IDENTIFIER]

Python

# Add hyperparameter tuning to the job config.
hyperparams = {
    'enableTrialEarlyStopping': True,
    'maxTrials': 30,
    'maxParallelTrials': 1,
    'resumePreviousJobId': [PREVIOUS_JOB_IDENTIFIER]}

# Add the hyperparameter specification to training inputs dictionary.
training_inputs['hyperparameters'] = hyperparams

# Build the job spec.
job_spec = {'jobId': my_job_name, 'trainingInput': training_inputs}

Hyperparameter tuning with Cloud TPU

It's recommended to use the eval metrics property in TPUEstimatorSpec if you're executing your hyperparameter tuning job with Cloud TPU on the AI Platform Training.

Frequently Asked Questions

Why is hyperparameter tuning important?

A machine learning model's behavior can largely be controlled via hyperparameter adjustment.

What is Keras?

The open-source software package known as Keras provides a Python interface for artificial neural networks. Keras provides the TensorFlow library interface.

What is a runtime error?

When a program is syntactically sound but has a bug only discovered during program execution, it is said to have a runtime error.

Conclusion

This blog has extensively discussed Using hyperparameter tuning. We hope this blog has helped you learn about how to use hyperparameter tuning. If you want to learn more, check out the excellent content on the Coding Ninjas Website:

Manage to serve accounts in GCP, Overview of hyperparameter tuning in GCP

Refer to our guided paths on the Coding Ninjas Studio platform to learn more about DSA, DBMS, Competitive Programming, Python, Java, JavaScript, etc. 

Refer to the links problems, top 100 SQL problems, resources, and mock tests to enhance your knowledge.

For placement preparations, visit interview experiences and interview bundle.

Do upvote our blog to help other ninjas grow. Happy Coding!