Do you think IIT Guwahati certified course can help you in your career?
Introduction
Saltstack, popularly known as salt, is a configuration management and orchestration tool. It helps in managing a large number of servers altogether. It is simple to use, fast, and can be easily manageable. The saltstack has the benefit of technologies and can also be written as plain modules.11zx
In this blog, we will learn about the reactor system in salt. We will further look at the types of reactions in the reactors and will thoroughly discuss the reactor through an example.
Reactor System in Salt
A salt reactor system triggers actions for a particular event in salt and responds to that event. For the event tags to match a given pattern and then to run the commands in response to those events matching the interface of the salt event bus is relatively easy.
To bind the SLS files to event tags on the master, this system is used. After the binding is done, the SLS files define the reactions that imply that the reactor system will have two parts:
The first is to set the reactor option in the master configuration file.
To define the reactions for the execution, these reactor files use the highdata like the state system.
Event System
To understand the reactors, knowing the event system is necessary and important. The event system has a local ZeroMQ PUB interface used to fire events.
Since this event is an open system, we use this event bus to send information and notify the salt and other systems information.
Every event has a tag, so with particular criteria, the event system fires the event. These tags are further allowed for fast top-level filtering of events. With the addition of the tag, each event also has an additional data structure associated with it.
The data structure that contains information about the events is a dictionary data structure.
How to Map Events to Reactor SLS files?
In the master configuration file, the event tags and reactor files are associated. By default, it is /etc/salt/master, or /etc/salt/master.d/reactor.conf.
The reactor support for salt:// file paths was added in version 2014.7.0.
Each event has a list of reactor SLS files that have to be run, and a list of event tags is present in the reactor of the master configuration section:
reactor: # Master config section "reactor"
- 'salt/minion/*/start': # Match tag "salt/minion/*/start"
- /srv/reactor/start.sls # when a minion starts things to do
- /srv/reactor/monitor.sls # Other things to do
- 'salt/cloud/*/destroyed': # to match tags globs are used
- /srv/reactor/destroy/*.sls # to match file names globs are used
- 'myco/custom/event/tag': # React to custom event tags
- salt://reactor/mycustom.sls # from the salt fileserver Reactor files can come
The above - salt://reactor/mycustom.sls refers to the base environment.
Use the querystring syntax (ex: (e.g. salt://reactor/mycustom.sls?saltenv=reactor) to pull this file from a different environment.
The reactor files are similar to the pillar SLS files and the state system. These are passed with the familiar context variables and are, by default, YAML + Jinja templates.
For further refining the reactions, we use the simple reactor files. When the minion's name is mysql1, or the name or the id in the event is mysql1, we define the following reaction. For the reactor system, we use the same compiler and data structure used in the state system.
The only difference is that the data must be matched up to the runner system and the salt command API. In the above example, for performing the highstate, the command that is being published to mysql1 with a function of state.apply
Similarly, for calling a runner, use the below code: This example will initiate execution of the runit orchestrator that is located at the /srv/salt/orchestrate/runit.sls and will also execute the orchestrate runner.
Lets us now look at the types of reactions that are available in the reactor system.
Name
Description
local
On the targeted minion, it runs a remote-execution function.
runner
To execute the runner function, use the runner reactor.
wheel
On the master, it executes a wheel function.
caller
On the masterless minion, it runs a remote-execution function.
In the future release, you can expect the naming of the local and caller reactions. They were not intuitively named, but these reaction types were named after the salt’s internal client interfaces. However, these reactions, i.e., local and caller will continue to work on the Reactor SLS files.
Location of Reactor SLS Files
The reactors' SLS files configured via the fileserver_backend configuration option can come from any of the backends enabled, and the files local to the master. As we do in state SLS files in the same way, the placed files can be referenced in the salt fileserver using a salt://URL.
To keep them organized, it is recommended that you place the orchestrate SLS and the reactor in their unique directory named subdirectories like orch/, orchestrate/, react/, reactor/, etc.
Writing Reactor SLS
Historically, different methods for passing arguments for the reactions are developed separately. The unified configuration schema gets applied to all the reaction types was newly introduced in the release of 2017.7.2.
Although the old configuration will be supported, there is yet to be a plan to replace it.
Local Reactions
We use the local reactions on the targeted minions to run a remote-execution function. In the old configuration, we required the user's keyword arguments and positional to be manually separated under the arg and kwarg arguments.
This could be more user-friendly, as it forces us to distinguish which arguments are what type and ensure that the positions arguments are ordered correctly. So if the master is running on a supported release, it is recommended to use the new config schema.
For running the below salt command, the above reaction will be equivalent.
salt -G 'kernel:Linux' state.single pkg.installed name=zsh fromrepo=updates
At the same indentation level as tgt you can pass the parameters in the LocalClient().cmdasync() method.
When the target expression that is defined in the tgt uses a target type that is other than the minion id glob, you are then required to use tgt_type.
In the release after 2017.7.0, the tgt_type argument was named expr_form.
Runner Reactions
To execute the runner function locally on the master, use the runner reactions. In the old config schema, to pass the arguments to the reactions directly, we used to pass the arguments directly under the name of the runner function. This further causes some unexpected issues with the reactors system's internal arguments. Also, you can pass the positional and keyword arguments to be manually get separated by the user under the arg and kwarg arguments, as we saw in the local reactions, but we also saw that this is not user-friendly. So if the master is running on the supported release, it is highly recommended that you use the new schema configuration.
The below two examples are equivalent to each other.
The reaction will be equivalent to running the following salt command if we assume that the tag event has foo and the data that get passed to the event is {'bar': 'baz'},
For running the wheel functions locally in the master, we use wheel reactions.
It is the same as we saw in the runner reactions that in the old config schema, we call the wheel reaction so that we can pass the arguments directly under the name of the wheel functions. You can also pass arguments in arg or kwarg arguments.
The below two examples are equivalent to each other.
We use the caller reactions on a minion daemon’s reactors system to run the remote-execution functions. It is essential to configure the reactor engine in the minion config file. Also, you must set up the watched events in a reactor section of the minion config file so that you can run the reactor in the minions.
The masterless minion uses this reactor, and this is the only way you can run the reactor in the masterless minion.
This involves passing arguments under the args parameter. This is available in both the config schemas, i.e., the old and the new ones. But the old schema only supports the positional arguments, so it is recommended to use the new schema if your masterless minion is running on the supported release.
The below two examples are equivalent to each other.
To run the below salt command, the above-salt reactions are equivalent:
salt-call file.touch name=/tmp/foo
Best Ways to Write the Reactor SLS Files
The work of the reactor is as follows:
For new events, the salt reactor watches the salt's event bus.
In the salt master config, under the reactor section, the event tag gets matched against all the event tags.
The data structure representing one or more function calls, the SLS files that match, are rendered into that data structure.
To a pool of worker threads, the data structure is given for the execution.
We match and render the reactor SLS files sequentially in a single process. And for this particular reason, it was important that the reactor SLS files must contain a few individual reactions. It does not support requisites, and the reactions are fired asynchronously, except for the caller.
The other reactions get piled up behind the current one because of complex jinja templating as it calls outs to slow the runner function and remote execution as these calls slow down the rendering. The worker pool is designed to handle complex, long-running, complex processes like orchestration jobs.
So orchestration is a natural fit when we have complex tasks in order. The orchestration SLS files are more complex and can also use requisites. We use orchestration to let the reactor system fire off the orchestration jobs, proceed with the other files, and perform complex tasks.
Jinja Context
Only minimal jinja context can be accessed by the reactor SLS files. The availability of grains and pillar is not available. To call the runner functions or the remote-execution salt object is made available, but you should use this for quick tasks and sparingly because of the reasons mentioned above:
The following variables have also been available in the Jinja Context in addition to the salt object.
The variables are
tag: That triggers the execution of the Reactor SLS file, the tag made from the event.
data: Data dictionary of the event
If the event is fired from a minion, and the data being passed to the event contains a data key, then the data dictionary will have an id key that will further include the minion id.
Advanced State System Capabilities
From the salt’s state system, the reactor SLS files by design don’t provide support to onlyif/unless conditionals, ordering, requisites, and the most powerful constructs.
Salt orchestrate system best performs the complex master-side operations, and also, to kick off the orchestrate run using the reactor is a very common parsing.
# /srv/salt/orchestrate/do_complex_thing.sls
{% set tag = salt.pillar.get('event_tag') %}
{% set data = salt.pillar.get('event_data') %}
#to a custom runner function pass the data from the event
#the function expects 'foo' argument
do_first_thing:
salt.runner:
- name: custom_runner.custom_function
- foo: {{ data.foo }}
# Wait for the runner to finish, then send execution to minions.
# Forward some data from the event down to the minion's state-run.
do_second_thing:
salt.state:
- tgt: {{ data.bar }}
- sls:
- do_thing_on_minion
- kwarg:
pillar:
baz: {{ data.baz }}
- require:
- salt: do_first_thing
Beacons and Reactors
When the data arrives at the master, an event gets initiated by the beacon, which will wrap it inside the second event.
Rather than the data, the object containing the beacon information will be the data{‘data}.
As for the events that get initiated directly on the event bus so rather than {{data[‘id’]}}, you will need to reference {{ data['data']['id'] }}.
Similarly, the data dictionary that gets attached to the event will be located in {{ data['data']['data'] }} instead of {{ data['data'] }}.
How do you Manually Fire an Event?
To manually fire an event, you have two options:
From the master
From the minion
Below we have briefly described how you should use the two options to fire an event.
From the Master
event.send runner is used.
salt-run event.send foo '{orchestrate: refresh}'
From the Minion
Call event.send, when you want to fire an event to the master from a minion call.
salt-call event.send foo '{orchestrate: refresh}'
Call event.fire when you want to fire an event to the minion’s local event bus:
salt-call event.fire '{orchestrate: refresh}' foo
Referencing Data Passed in the Events
Let’s assume the above examples: The reactor SLS gets triggered and executed with {{ data['data']['orchestrate'] }} that is equal to 'refresh' when it watches the event tag foo.
Get Information about Events
To see what event is fired and what data is available in each event, use the state.event.runner.
You get the best window into the reactor when you run the master with debugging enabled, and also it should run in the foreground.
When the master sees the event, the output will contain the following information:
The rendered SLS file (or any errors that get generated when you render the SLS file).
The response of the master to that event.
You should stop the master.
Now start the master manually.
salt-master -l debug
3. The next is to look at the log entries in the form:
[DEBUG] for tag foo/bar gatering the reactors
[DEBUG} for tag foo/bar compiling the reactions
[DEBUG] From the file:/path/to/the/reactor_file.sls gets rendered data
<... Rendered output appears here. ...>
Doing Jinja parsing, you get the rendered output, which is an excellent way to view the result of referencing the Jinja variables. The reactor will ignore when the result is empty, and jinja produces an empty result.
Passing Event Data to Minions or Orchestration as Pillar
Since both the functions take a keyword argument named pillar, you must pass the data as an inline pillar to easily pass the data from the reactor SLS to the state.apply.
In the below example, we use salt reactors to hear about the event that gets fired when on the master using salt-key. The key for a new minion is accepted.
Via inline pillar, the reactor fires a state.apply command that targets the Haproxy servers and passes the new minion's ID to the state file from the event.
The data is available in the state file using the normal pillar lookup syntax. From the Salt Mine, the following example will grab the web server's name and the IP address. When you invoke this state from the reactor then, you get the custom pillar value, and the minion will also be added to the pool. To ensure that the HAProxy won’t get the traffic, you get this with a disabled flag.
/srv/salt/haproxy/refresh_pool.sls\
{% set new_minion = salt['pillar.get']('new_minion') %}
listen web *:80
balance source
{% for server,ip in salt['mine.get']('web*', 'network.interfaces', ['eth0']).items() %}
{% if server == new_minion %}
server {{ server }} {{ ip }}:80 disabled
{% else %}
server {{ server }} {{ ip }}:80 check
{% endif %}
{% endfor %}
A Complete Example
Let’s thoroughly go through an example.
In this example, we will assume that we have a group of servers that accept keys automatically and will also come online at random. Further, we will add the constraint that we don’t want all the servers to get accepted automatically.
And for this, let us go through this example. In this, we assume that we have all hosts that consist of an id that starts with ink, will get automatically accepted, and also will execute state.apply. We will also add here that the host that was replaced (i.e., new key) will get accepted.
For this, our master configuration file will be simple. The minions that attempt the authentication will match the tag of salt/auth. When the minion key gets accepted, we will add a more refined tag that will further include the minion id, which will later be helpful in matching.
If the key is rejected, the minion process will die in this SLS file. Also, when the key gets rejected, we will delete the key on the master and will tell the master to ssh into the minion and will ensure to tell to restart the minion. Further, when the id stats with ink and key is pending, we will accept that key. By default, after every ten seconds, the minion waiting on the pending key will retry for authentication.
/srv/reactor/auth-pending.sls:
{# Ink server failed to authenticate -- remove accepted key #}
{% if not data['result'] and data['id'].startswith('ink') %}
minion_remove:
wheel.key.delete:
- args:
- match: {{ data['id'] }}
minion_rejoin:
local.cmd.run:
- tgt: salt-master.domain.tld
- args:
- cmd: ssh -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no "{{ data['id'] }}" 'sleep 10 && /etc/init.d/salt-minion restart'
{% endif %}
{# Ink server is sending new key -- accept this key #}
{% if 'act' in data and data['act'] == 'pend' and data['id'].startswith('ink') %}
minion_add:
wheel.key.accept:
- args:
- match: {{ data['id'] }}
{% endif %}
Statements are not needed as we have already limited this action. To Ink the servers in the master configuration, we have limited this action
/srv/reactor/auth-complete.sls:
{# When an Ink server connects, run state.apply. #}
highstate_run:
local.state.apply:
- tgt: {{ data['id'] }}
- ret: smtp
Using the smtp_return returner, the above will also return the highstate result data. For this to work, the returner must get configured.
Syncing Custom Types on Minion Start
On every highstate, salt sync all custom types by running a saltutil.sync_all.
When the top file is first compiled, a minion will not yet have custom types synced. There is an issue with the initial highstate, i.e., the chicken and egg issue.
When the minion first starts up and connects to the master, each minion fires an event salt/minion/*/start, and a simple reactor watches this event, and this can be worked out. Create /srv/reactor/sync_grains.sls on the master with the below contents:
When it starts, it will cause the master to instruct each minion to sync its custom grains, so when the highstate is executed this will make these grains available.
To sync other types, you must replace the local.saltutil.sync_grains with local.saltutil.sync_all, local.saltutil.sync_modules, or whatever suits the intended use case.
Also, when not every minion should get synced on startup, you must replace the * with a different glob. This will further narrow down the set of minions matching the reactor.
Ex: This will salt/minion/appsrv*/start only match the minion's IDs beginning with apparv.
Reactor Tuning for Large-Scale Installations
Inside salt.utils.process.ThreadPool reactors use thread-pool implementations. The jobs picked by standard python threads use python’s stdlib queue to enqueue jobs.
On the thread pool, by the firing method, false is returned when the queue is full.
For the selection of proper values for the reactor, we have a few other comments:
For the reactor, there are things to say about selecting the proper values. reactor_worker_hwm should be increased or even set to 0 to bind it only by the available memory. This should be done in a situation where we are expecting that reactors will execute many-long running jobs.
If your concern is performance, you can also increase the value for the reactor_worker_threads. This should be done when you expect many long-running jobs and execution concurrency. By increasing the value, you will be able to control the number of concurrent threads pulling the jobs from the queue and executing them. This for sure depends on the relationship to the speed at which the queue itself gets filled up. Each thread containing a copy of the salt code required to perform the desired action is the price you pay for the value.
Frequently Asked Questions
What is saltstack?
Salt is a configuration management and remote execution tool that helps execute commands on the remote node. It is simple to use, fast, and can be easily manageable.
Where do we use saltstack?
The saltstack is an orchestration tool that helps change existing systems. It helps easily install the software in the IT environment and helps manage thousands of servers in a single go.
Is saltstack still free of cost to developers?
Saltstack is a free, open-source download and is free of cost to the programmers; however, their enterprise version costs $150 per machine per year.
What are the types of reactions that are available in the reactor system?
The reactions available in the reactor system are local, runner, wheel, and caller.
What is a reactor system in salt?
A salt reactor system triggers actions for a particular event in salt and responds to that event. For the event tags to match a given pattern and then to run the commands in response to those events matching the interface of the salt event bus is quite easy.
Conclusion
In this blog, we thoroughly discussed the reactor system in salt. We also learn about the types of reactions in the reactor, debugging the reactor, and many other important concepts of the reactor system in salt.
16+ registered 
