Table of contents
1.
Introduction 
2.
What is Orchestrate Runner?
3.
Writing SLS Files
4.
How to Execute the Orchestrate Runner?
5.
Masterless Orchestration
6.
Examples
6.1.
Function  
6.1.1.
Fail Functions
6.2.
STATE
6.3.
HIGHSTATE
6.4.
Runner
7.
Return Codes in Runner/ Wheel Jobs
8.
More Complex Orchestration
9.
Parsing Results Programmatically
10.
Running States on the Master Without a Minion
11.
Limitations
12.
Frequently Asked Questions
12.1.
What is saltstack?
12.2.
Where do we use saltstack?
12.3.
Is saltstack still free of cost to developers?
12.4.
What is Orchestrate Runner in salt?
12.5.
What is the return output from the orchestration jobs?
13.
Conclusion
Last Updated: Mar 27, 2024
Medium

Orchestrate Runner in Salt

Author Gunjan Batra
0 upvote
Career growth poll
Do you think IIT Guwahati certified course can help you in your career?

Introduction 

Saltstack, popularly known as salt, is a configuration management and orchestration tool. It helps in managing a large number of servers altogether. It is simple to use, fast, and can be easily manageable. The saltstack has the benefit of technologies and can also be written as plain modules.

Saltstack

In this blog, we will learn about the Orchestrate Runner in Salt. We will further look at the masterless orchestrate, SSL files, and the way to parse the results of the orchestrate runner. 

What is Orchestrate Runner?

Configuring large amounts of minions at once is quite difficult. Orchestration will be helpful to you for configuring all the minions at once. Orchestrate Runner in salt helps in controlling the activities of the minions.

What is Orchestrate Runner?

To update and manage systems, configuration management relies on orchestration technology.  In Salt 2015.8.0, the orchestrate runner replaced the OverState System.

Orchestrate Runner is officially called state.sls.runner. This offers you all the functionality of the overstate but with the added advantage. 

The advantages are:

  • All the global state arguments and requisites available in the states can now be used with the orchestrate runner. 
     
  • These functions/States will also work on salt-ssh minions. 
     

For example: In front of a cluster of web servers, when you want to set up the load balancer, you can ensure that first, you set up the load balancer, and then the same will match with the configuration that will be applied for the whole cluster.

The salt state system can be generalized to a salt master context with the help of orchestrate runner. On the other hand, the state.sls, state.highstate et al.functions are concurrent and independent on each salt-minion, whereas the salt.orchestrate runner gets executed on the master, and it provides you with a master-level view. also, through this, you can control the requisites like state ordering and conditionals. It helps further to order the application of states that must not happen simultaneously on different minions, or if a minion fails one of its states, it can halter the state for running on all minions. 

The state.sls, state.highstate et al.functions allow you the feature to manage each minion and to statefully manage the entire infrastructure you can use the state.orchestrate runners.

Writing SLS Files

SLS files are the core of the salt state system. It further represents which state should be in, and it is a setup that contains the data in a simple format. SLS files are also called configuration management, as it manages the configuration of the files and looks at the setup of those files. 

The orchestrate SLS files have the same location as the State SSL files. The SSL files available to the reactor and orchestrator get impacted by both the file_roots and gifts_remotes. To prevent this confusion and duplicate naming, it is recommended that you keep the orchestrator SSL and reactor files in their unique named subdirectories, which are _orch/, orch/, _orchestrate/, react/, _reactor/, etc.

How to Execute the Orchestrate Runner?

The synonym for state.orchestrate is state.orch

To avoid confusion with the salt.sls execution, the runner function was renamed to state.orchestrate, but you must use the state.sls execution function in the versions from 0.17.0  through 2014.1.0. 

How to Execute the Orchestrate Runner?

The command format for orchestrate runner is the same as for the salt.sls function, but since orchestrate is a runner; it is executed with the salt-run command rather than the salt. 

For better understanding, let us go through the following example:

Run on the master the following command to apply the states defined in the state.sls file called /srv/salt/orch/webserver.sls. 

salt-run state.orchestrate orch.webserver

Masterless Orchestration

This is available in the new version 2016.11.0.  

In an SLS file, masterless orchestration support only salt.state command, but currently, it does not support the salt.function command.

Masterless Orchestration

The orchestrate runner is available as an execution module to support salt orchestration on the masterclass minion. Masterclass orchestration syntax is the same, but it uses the salt-call command, and the minion configuration must contain the file_mode: local option. 

On the command line, you can alternatively use salt-call - - local. 

salt-call --local state.orchestrate orch.webserver

Let’s now understand the examples below: 

Examples

Examples

Function  

To execute a function, use salt.function 

# /srv/salt/orch/cleanfoo.sls
cmd.run:
  salt.function:
    - tgt: '*'
    - arg:
      - rm -rf /tmp/foo
salt-run state.orchestrate orch.cleanfoo


The state ID will be the default name when you omit the “name” argument or in the salt.function the execution module to run. To avoid conflicting IDs, you can specify the “argument”.

copy_some_file:
  salt.function:
    - name: file.copy
    - tgt: '*'
    - arg:
      - /path/to/file
      - /tmp/copy_of_file
    - kwarg:
        remove_existing: true


Fail Functions

You can also write fail functions as a custom execution module. They accept a single argument and return a boolean value.

You must sync the fail functions using salt-run saltutil.sync_modules as fail functions run on the master.

There can be a case when the function doesn’t set a return code. Still, when you run a remote execution function in the orchestration, certain return values for the function might indicate failure. In such circumstances, using a fail function allows easy access to success or failure.

The example below is how fail functions work:

def check_func_result(input):
    if some_condition:
        return True
    else:
        return False


In Orchestration SLS, the function above can be referenced, and it is like as:

do_stuff:
  salt.function:
    - name: modname.funcname
    - tgt: '*'
    - fail_function: mymod.check_func_result

STATE

For executing the state, use the salt.state.

# /srv/salt/orch/webserver.sls
install_nginx:
  salt.state:
    - tgt: 'web*'
    - sls:
      - nginx
salt-run state.orchestrate orch.webserver

HIGHSTATE

In your state configuration, to run the highstate, set the highstate : True. 

# /srv/salt/orch/web_setup.sls
webserver_setup:
  salt.state:
    - tgt: 'web*'
    - highstate: True
salt-run state.orchestrate orch.web_setup


Runner

Use salt.runner to execute another runner. Let’s understand more by this example.

# /srv/salt/orch/deploy.sls
create_instance:
  salt.runner:
    - name: cloud.profile
    - prof: cloud-centos
    - provider: cloud
    - instances:
      - server1
    - opts:
        minion:
          master: master1


Use jinja variables and inline pillar data to get a more dynamic state. The state will look like this in the same above example by passing the pillar data.

# /srv/salt/orch/deploy.sls
{% set servers = salt['pillar.get']('servers', 'test') %}
{% set master = salt['pillar.get']('master', 'salt') %}
create_instance:
  salt.runner:
    - name: cloud.profile
    - prof: cloud-centos
    - provider: cloud
    - instances:
      - {{ servers }}
    - opts:
        minion:
          master: {{ master }}


To execute with pillar data, use the below command:

salt-run state.orch orch.deploy pillar='{"servers": "newsystem1",
"master": "mymaster"}'

Return Codes in Runner/ Wheel Jobs

Via State-run Dictionary state jobs are able to report failure. In the __context__ dictionary,  by setting a recode key remote execution jobs are able to report failure. However, the case is different with runner and wheel jobs. In this, runner (salt.runner) and wheel (salt.wheel) will only report the failure when these functions will raise an exception. 

Just as you can do in remote execution functions now with the 2018.3.0 release, you can now set the retcode in the runner and wheel functions.

Below is an example of the pseudocode for the following:

def myrunner():
    ...
    # do stuff
    ...
    if some_error_condition:
        __context__["retcode"] = 1
    return result.


With this, you can allow the custom wheel/runner to report the failure so that the requisites can accurately inform you that the job has failed.

More Complex Orchestration

To easily configure complex orchestration tools, You must configure many states/functions in a single file, which later, combined with the full suite of requisites and other global state arguments, will further help set up the configuration.

Until and unless you prevent the order by any requisites and other global state arguments,  the states/functions that are defined, get executed in their respective order. The same has been in the SLS files since 0.17.0.

bootstrap_servers:
  salt.function:
    - name: cmd.run
    - tgt: 10.0.0.0/24
    - tgt_type: ipcidr
    - arg:
      - bootstrap


storage_setup:
  salt.state:
    - tgt: 'role:storage'
    - tgt_type: grain
    - sls: ceph
    - require:
      - salt: webserver_setup


webserver_setup:
  salt.state:
    - tgt: 'web*'
    - highstate: True


The orchestration will be carried out as follows from the above setup:

  1. In the 10.0.0.0/20 subnet on all the minions, the shell command bootstrap will be executed. 
     
  2. As the storage-setup state requires it, the minions that start with the ID “web”, highstate will be running on all those minions. 
     
  3. All the minions with a grain called role with the value of storage the ceph SLS target gets executed. 

    The salt run is always executed on the master. 

Parsing Results Programmatically

The return output of Orchestration jobs comes in a particular data structure. Depending on the outputter used, the data structure gets represented differently. When you use a default outputter, the output you receive is in a human-readable format. 

Program

Let’s see the below orchestration SLS:

good_state:
  salt.state:
    - tgt: myminion
    - sls:
    - succeed_with_changes


bad_state:
  salt.state:
    - tgt: myminion
    - sls:
    - fail_with_changes


mymod.myfunc:
  salt.function:
    - tgt: myminion


mymod.myfunc_false_result:
  salt.function:
    - tgt: myminion


The above orchestration SLS will produce the output when you will use the default outputter. 
fa5944a73aa8_master:
----------
          ID: good_state
    Function: salt.state
      Result: True
     Comment: States ran successfully. Updating myminion.
     Started: 21:08:02.681604
    Duration: 265.565 ms
     Changes:
              myminion:
              ----------
                        ID: test succeed with changes
                  Function: test.succeed_with_changes
                    Result: True
                   Comment: Success!
                   Started: 21:08:02.835893
                  Duration: 0.375 ms
                   Changes:
                            ----------
                            testing:
                                ----------
                                new:
                                    Something pretended to change
                                old:
                                    Unchanged


              Summary for myminion
              ------------
              Succeeded: 1 (changed=1)
              Failed:    0
              ------------
              Total states run:     1
              Total run time:   0.375 ms
----------
          ID: bad_state
    Function: salt.state
      Result: False
     Comment: Run failed on minions: myminion
     Started: 21:08:02.947702
    Duration: 177.01 ms
     Changes:
              myminion:
              ----------
                        ID: test fail with changes
                  Function: test.fail_with_changes
                    Result: False
                   Comment: Failure!
                   Started: 21:08:03.116634
                  Duration: 0.502 ms
                   Changes:
                            ----------
                            testing:
                                ----------
                                new:
                                    Something pretended to change
                                old:
                                    Unchanged


              Summary for myminion
              ------------
              Succeeded: 0 (changed=1)
              Failed:    1
              ------------
              Total states run:     1
              Total run time:   0.502 ms
----------
          ID: mymod.myfunc
    Function: salt.function
      Result: True
     Comment: Function ran successfully. Function mymod.myfunc ran on myminion.
     Started: 21:08:03.125011
    Duration: 159.488 ms
     Changes:
              myminion:
                  True
----------
          ID: mymod.myfunc_false_result
    Function: salt.function
      Result: False
     Comment: Running function mymod.myfunc_false_result failed on minions: myminion. Function mymod.myfunc_false_result ran on myminion.
     Started: 21:08:03.285148
    Duration: 176.787 ms
     Changes:
              myminion:
                  False


Summary for fa5944a73aa8_master
------------
Succeeded: 2 (changed=4)
Failed:    2
------------
Total states run:     4
Total run time: 778.850 ms


To get the output in a parsable and easily loadable format, use the json outputter. 

salt-run state.orchestrate test --out=json
{
    "outputter": "highstate",
    "data": {
        "fa5944a73aa8_master": {
            "salt_|-good_state_|-good_state_|-state": {
                "comment": "States ran successfully. Updating myminion.",
                "name": "good_state",
                "start_time": "21:35:16.868345",
                "result": true,
                "duration": 267.299,
                "__run_num__": 0,
                "__jid__": "20171130213516897392",
                "__sls__": "test",
                "changes": {
                    "ret": {
                        "myminion": {
                            "test_|-test succeed with changes_|-test succeed with changes_|-succeed_with_changes": {
                                "comment": "Success!",
                                "name": "test succeed with changes",
                                "start_time": "21:35:17.022592",
                                "result": true,
                                "duration": 0.362,
                                "__run_num__": 0,
                                "__sls__": "succeed_with_changes",
                                "changes": {
                                    "testing": {
                                        "new": "Something pretended to change",
                                        "old": "Unchanged"
                                    }
                                },
                                "__id__": "test succeed with changes"
                            }
                        }
                    },
                    "out": "highstate"
                },
                "__id__": "good_state"
            },
            "salt_|-bad_state_|-bad_state_|-state": {
                "comment": "Run failed on minions: test",
                "name": "bad_state",
                "start_time": "21:35:17.136511",
                "result": false,
                "duration": 197.635,
                "__run_num__": 1,
                "__jid__": "20171130213517202203",
                "__sls__": "test",
                "changes": {
                    "ret": {
                        "myminion": {
                            "test_|-test fail with changes_|-test fail with changes_|-fail_with_changes": {
                                "comment": "Failure!",
                                "name": "test fail with changes",
                                "start_time": "21:35:17.326268",
                                "result": false,
                                "duration": 0.509,
                                "__run_num__": 0,
                                "__sls__": "fail_with_changes",
                                "changes": {
                                    "testing": {
                                        "new": "Something pretended to change",
                                        "old": "Unchanged"
                                    }
                                },
                                "__id__": "test fail with changes"
                            }
                        }
                    },
                    "out": "highstate"
                },
                "__id__": "bad_state"
            },
            "salt_|-mymod.myfunc_|-mymod.myfunc_|-function": {
                "comment": "Function ran successfully. Function mymod.myfunc ran on myminion.",
                "name": "mymod.myfunc",
                "start_time": "21:35:17.334373",
                "result": true,
                "duration": 151.716,
                "__run_num__": 2,
                "__jid__": "20171130213517361706",
                "__sls__": "test",
                "changes": {
                    "ret": {
                        "myminion": true
                    },
                    "out": "highstate"
                },
                "__id__": "mymod.myfunc"
            },
            "salt_|-mymod.myfunc_false_result-mymod.myfunc_false_result-function": {
                "comment": "Running function mymod.myfunc_false_result failed on minions: myminion. Function mymod.myfunc_false_result ran on myminion.",
                "name": "mymod.myfunc_false_result",
                "start_time": "21:35:17.486625",
                "result": false,
                "duration": 174.241,
                "__run_num__": 3,
                "__jid__": "20171130213517536270",
                "__sls__": "test",
                "changes": {
                    "ret": {
                        "myminion": false
                    },
                    "out": "highstate"
                },
                "__id__": "mymod.myfunc_false_result"
            }
        }
    },
    "retcode": 1
}


To make the data easy and more accurate, the 2018.3.0 release included a couple of fixes.

The first change is you will be able to set the return code in the wheel function or custom runner. The second change is the return data that how failures get included in the data. 

Before the 2018.3.0 release, the minions that get failed in a salt.state orchestration would show up in the comment field of the return data. The output shown was not in the human-readable format that could not be easily parsed. Now, these are included in the changes dictionary, along with the minions that succeed. The salt.state jobs now also handle the failure similar to the salt.functions jobs that failed because of the fail functions returned false. 

Running States on the Master Without a Minion

To execute states on the master without using a minion, you can use orchestrate runner. 

Let’s understand this through an example.

Assume that salt://foo.sls contains the following SLS:

/etc/foo.conf:
  file.managed:
    - source: salt://files/foo.conf
    - mode: 0600


In such cases, running salt-run state.orchestrate foo would also be equal to running a state.sls foo. It will only execute on the master and will not require a minion daemon to run on the master. 

On a technical basis, it is not the orchestration, but it is useful in many other use cases. 

Limitations

Using this method, only one SLS target will be running at a time, whereas using the state.sls gives you the ability to use multiple SLS files using a comma-separated list.

Frequently Asked Questions

What is saltstack?

Salt is a configuration management and remote execution tool that helps execute commands on the remote node. It is simple to use, fast, and can be easily manageable. 

Where do we use saltstack?

The saltstack is an orchestration tool that helps change existing systems. It helps easily install the software in the IT environment and helps manage thousands of servers in a single go. 

Is saltstack still free of cost to developers?

Saltstack is a free, open-source download and is free of cost to the programmers; however, their enterprise version costs $150 per machine per year. 

What is Orchestrate Runner in salt?

Orchestrate Runner is officially called state.sls.runner. The orchestrate runner offers you all the functionality of the overstate but with the added advantage. The advantages are that all the global state arguments and requisites available in the states can now be used with the orchestrate runner, and these functions/States will also work on salt-ssh minions. 

What is the return output from the orchestration jobs?

The return output of Orchestration jobs comes in a particular data structure. Depending on the outputter used, the data structure gets represented differently. When you use a default outputter, the output you receive is in a human-readable format. 

Conclusion

In this article, we have discussed the orchestrate runner in salt. We further looked at masterless orchestration and examples and the way by which we can parse the results of the orchestration jobs. In the end, we looked at the limitations of the orchestrate runner. 

To learn more about salt, please refer to blogs.

About Salt Engine

Target Minions in Salt

About Salt Runners

Salt Event System

Refer to our guided paths on Coding Ninjas Studio to learn more about DSA, Competitive Programming, JavaScript, System Design, etc. Enrol in our coursesrefer to the mock test and problems look at the interview experiences and interview bundle for placement preparations.

Happy Coding!

Live masterclass