Code360 powered by Coding Ninjas X Naukri.com. Code360 powered by Coding Ninjas X Naukri.com
Table of contents
1.
Introduction
2.
Workflow Basics
2.1.
Creating Workflows
2.2.
Executing Workflows
2.3.
Passing Runtime arguments
2.4.
Scheduling Workflows
3.
Advanced
3.1.
Workflow Syntax
3.2.
Standard library
3.3.
Workflows execution APIs
3.4.
Connectors
4.
Frequently Asked Questions
4.1.
What is the structure of a Workflow source file?
4.2.
What are the different IAM roles available for workflows?
4.3.
How to control the order of execution in a Workflow?
5.
Conclusion
Last Updated: Mar 27, 2024

Overview of Workflows

Author Yashesvinee V
0 upvote
Leveraging ChatGPT - GenAI as a Microsoft Data Expert
Speaker
Prerita Agarwal
Data Specialist @
23 Jul, 2024 @ 01:30 PM

Introduction

A workflow manages repetitive processes and tasks that occur in a particular order. In GCP,  Workflows is a fully-managed orchestration platform that executes services in a specific order. These workflows combine custom services hosted on other Google Cloud services and APIs. Workflows are serverless, can be easily scaled, and incurs no charges when idle. Since workflows contain no code or dependencies, it does not require security patches. They can reliably execute without maintenance. A workflow can hold state and wait for up to a year. Let us cover some of the basic and advanced concepts of Workflow in GCP.

Workflow Basics

Workflows support various use cases, including IT process automation, business processes, batch jobs and Service orchestration. A workflow consists of a series of steps as described in the Workflow syntax. This is written in either YAML or JSON and is known as the workflow's definition. Deploying a workflow makes it ready for execution. An execution involves running the logic contained in a workflow's definition.

Creating Workflows

Step 1: Enable the Workflows API.

Step 2: Create a Service account and assign the necessary roles to create and manage workflows.

Step 3: Go to the Workflows page and select Create.

Step 4: Enter the name for a new workflow, and select an appropriate region and a service account. Click on Next.

Step 5: Enter the Workflow definition in the editor and click on Deploy.

Select the workflow's name on the Workflows page to update an existing workflow. Navigate to the Edit Workflow page and make the required changes. Note that the workflow's name can not be edited. Click on Next and the Deploy.

Executing Workflows

Users can execute a workflow using the client libraries using the Google Cloud CLI or by sending a request to the Workflows REST API. To execute using REST APIs, follow the steps:

Step 1: Get the list of workflows in the terminal and find the one required to be executed.

Step 2: Run the following command to execute the workflow.

curl \
--request POST \
--header "Authorization: Bearer "$(gcloud auth application-default print-access-token) \
--header 'Content-Type: application/json' \
"https://workflowexecutions.googleapis.com/v1/projects/PROJECTID/locations/WORKFLOWREGION/workflows/WORKFLOWNAME/executions"

If the workflow takes additional arguments, include the following flag to the command.

--data '{"argument":"{\"PARAMETER\":\"VALUE\"}"}'

Executing a workflow runs the workflow definition associated with the workflow. After execution, the history and results are retained for a limited time. We can retrieve the execution attempts and ID by running this command.

gcloud workflows executions list WORKFLOW_NAME

The following command is used to check the status of an execution attempt.

gcloud workflows executions wait EXECUTION_ID

The status of the last execution attempt can be retrieved using this command.

gcloud workflows executions describe-last

Passing Runtime arguments

We can pass runtime arguments in a workflow execution request and access them using workflow variables.

Step 1: Create a new workflow or choose an existing one from the Workflows page.

Step 2: Add a params field in the workflow definition as shown.

main:
  params: [ARG_NAME]
  steps:
    ...

Step 3: Deploy the workflow.

Scheduling Workflows

Cloud Scheduler is used to run workflows on a particular schedule. This can be done from the Google Cloud Console or in the CLI.

Step 1: Enable the Cloud Scheduler API.

Step 2: Create a service account to make requests to the Workflow API. Assign the account a workflows.invoker role to permit triggering a workflow.

Step 3: Create a scheduler job to trigger the workflow. 

Step 4: Go to the Edit Workflow page and click Add new Trigger > Scheduler.

Step 5:  Define the schedule and configure the execution. Click on Create.

Get the tech career you deserve, faster!
Connect with our expert counsellors to understand how to hack your way to success
User rating 4.7/5
1:1 doubt support
95% placement record
Akash Pal
Senior Software Engineer
326% Hike After Job Bootcamp
Himanshu Gusain
Programmer Analyst
32 LPA After Job Bootcamp
After Job
Bootcamp

Advanced

Workflows to execute services in a defined order. Workflows require no infrastructure management and can be scaled seamlessly with demand, including scaling down to zero.

Workflow Syntax

The Workflow syntax defines the desired steps and order of execution. 

  • Every workflow must have at least one step. 
     
  • The step name can contain alphanumeric characters as well as underscores. 
     
  • The main workflow is placed in the “main” block. 
     
  • By default, the workflow steps are executed in the order they appear in the definition.
     
  • A “steps” block is used to nest a series of steps and further define the workflow. Variables declared in a steps block can be accessed outside of the block.
     

Here is a simple example:

main:
      steps:
      - series_one:
          steps:
              - step_a:
                  call: http.get
                  args:
                      url: https://host.com/... 
                  result: api_response1
              - step_b:
                  assign:
                      - varA: "A"
                      - varB: "B"

Standard library

The Workflows standard library allows users to construct arguments for service and process responses easily. There is no need to import or load libraries in a workflow. Expression helpers are used for data type and format conversions. Following are some of the functions present in the standard library.

Workflow Standard Library Functions

Workflows execution APIs

Workflowexecutions.googleapis.com is a service used to execute workflows created with Workflows API. It is recommended to use Google-provided client libraries to call this service. The service provides two primary discovery documents, a machine-readable specification for describing and consuming REST APIs. These documents can be used to build client libraries and plugins that interact with Google APIs.

A service endpoint specifies the network address of the API service. A service can have multiple endpoints. The endpoint for the mentioned service is “https://workflowexecutions.googleapis.com”.

Connectors

Workflow connectors make accessing other Google Cloud products easier within a workflow. Invoking a connector is similar to invoking an HTTP endpoint. A connector call uses the call and args fields. A timeout value and polling policy are set in the connector_params field. Here are some of the connectors supported by workflows.

Workflow Connectors

Frequently Asked Questions

What is the structure of a Workflow source file?

A workflow source file contains only one main workflow, which might have sub workflows. Workflow definitions are stored as a YAML or JSON file. Code samples are provided in both YAML and the equivalent JSON.

What are the different IAM roles available for workflows?

Workflows Admin has full access to all workflows. Workflows Editor has read-write access to all workflows. Workflows Invoker can access and manage the execution of workflows. Workflows Viewer has read-only access to the workflows.

How to control the order of execution in a Workflow?

We can use jumps or for loops to control the order of executing the workflow's steps. Basic jumps help define which step the workflow runs next. Conditional jumps are built on basic jumps and allow using conditional expressions to control the order of execution through a workflow.

Conclusion

This blog gives a brief overview of Workflows in Google Cloud. It covers the basics of creating, executing and scheduling workflows on GCP. It also explains the workflow syntax, the standard library and connectors related to workflows. 

Check out our articles on Cloud Logging in GCP, Monitoring Agent and Identity Access ManagementExplore our Library on Coding Ninjas Studio to gain knowledge on Data Structures and Algorithms, Machine Learning, Deep Learning, Cloud Computing and many more! Test your coding skills by solving our test series and participating in the contests hosted on Coding Ninjas Studio! 

Looking for questions from tech giants like Amazon, Microsoft, Uber, etc.? Look at the problems, interview experiences, and interview bundle for placement preparations.

Upvote our blogs if you find them insightful and engaging! Happy Coding!

Thank you
 

Live masterclass