High availability in Sentinel in redis

Sentinel quick start

Obtaining Sentinel

Sentinel 2 is the most recent version of Sentinel. It's a rework of the original Sentinel implementation that uses more powerful and predictable algorithms.

Since Redis 2.8, a stable release of Redis Sentinel has been available.

New features are sometimes backported into the latest stable branch as soon as they are considered stable, and new developments are done in the unstable branch.

Version 1 of Redis Sentinel, which came with Redis 2.6, is deprecated and should not be utilized.

Fundamental things to know about Sentinel before deploying

1. A robust deployment requires at least three Sentinel instances.
2. The three Sentinel instances should be installed on systems or virtual machines that are thought to fail in their own right. For example, multiple physical servers or Virtual Machines could be running in various availability zones.
3. Because Redis uses asynchronous replication, the Sentinel + Redis distributed system cannot guarantee that acknowledged writes are kept during failures. However, some approaches to implementing Sentinel limit the window in which writes can be lost to specific times, while others are less safe.
4. Your clients must have Sentinel support. Some popular client libraries support Sentinel, but not all.
5. No HA configuration is safe if you don't test it in development environments or, better yet, in production environments to see if it works. You might have a misconfiguration that you won't notice until it's too late (at 3 am when your master stops working).
6. Sentinel, Docker, and other types of Network Address Translation and Port Mapping should be used with caution: Sentinel auto-detection of other Sentinel processes and the list of replicas for a master is broken when Docker conducts port remapping. For more details, see the section on Sentinel and Docker later in this text.

Sentinel, Docker, NAT, and possible issues

Docker employs a port mapping mechanism, meaning that programs running within Docker containers may be exposed to a different port than the program believes it is using. This is useful for running numerous containers in the same server using the same ports simultaneously.

There are alternative Network Address Translation arrangements where ports may be remapped, and sometimes not just ports but also IP addresses.

Sentinel has two difficulties when ports and addresses are remapped:

1. Sentinel auto-discovery of other Sentinels is no longer possible since it relies on hello messages in which each Sentinel declares which port and IP address it is listening for connections. Sentinels, on the other hand, have no way of knowing whether an address or port has been remapped, therefore, they are broadcasting incorrect information for other Sentinels to connect to.
2. The address is identified by the master inspecting the distant peer of the TCP connection. The replica itself declares the port during the handshake; nevertheless, it may be incorrect for the same reason as described in point 1.

Obtaining the address of the current master

Sentinel also serves as a configuration provider for clients who want to connect to a group of masters and replicas, as previously stated. Clients do not know who is the currently active master for a set of instances due to possible failovers or reconfigurations.

Sentinel API

Sentinel has an API for inspecting its state, checking the health of monitored masters and replicas, subscribing to specific notifications, and changing the Sentinel configuration at runtime.

Sentinel uses TCP port 26379 by default (note that 6379 is the normal Redis port). Sentinels receive commands via the Redis protocol, thus, you may communicate with them using redis-cli or any other unmodified Redis client.

You can directly query a Sentinel to see what the state of the monitored Redis instances is from its perspective and what other Sentinels it knows. Alternatively, you can use Pub/Sub to receive push notifications from Sentinels whenever an event occurs, such as a failover or an instance entering an error condition, and so on.

Replicas priority

A configuration parameter named replica-priority exists in Redis instances. Sentinel uses this information, which is exposed by Redis replica instances in their INFO output, to choose a replica from the ones that can failover a master:

• The replica is never promoted to master if the replica priority is set to 0.
• Sentinel prefers replicas with a lower priority number.

If a replica S1 is located in the same data center as the current master and another replica S2 is located in a different data center, S1 can be given a priority of 10, and S2 can be given a priority of 100, and both S1 and S2 are available, S1 will be favored.

Sentinel and Redis authentication

When the master is configured to require client authentication, replicas must know the credentials to authenticate with the master and establish the master-replica connection required for the asynchronous replication protocol.

Configuring Sentinel instances with authentication

Clients can be required to authenticate with the AUTH command to access Sentinel instances. The Access Control List (ACL) is accessible starting with Redis 6.2, whereas previous versions (starting with Redis 5.0.1) only offer password-based authentication.

Sentinel's authentication settings should be applied to each of your deployment's instances, and each instance should utilize the same configuration. Additionally, ACL and password-only authentication should not be combined.

Sentinel Access Control List authentication

The first step in using ACL to secure a Sentinel instance prohibits unauthorized access. To do so, disable the default superuser (or at the very least set a strong password for it), create a new one, and provide it access to Pub/Sub channels:

127.0.0.1:5000> ACL SETUSER admin ON >admin-password allchannels +@all

OK

127.0.0.1:5000> ACL SETUSER default off

OK

Sentinel connects to other instances using the default user. With the following configuration directives, you can provide the credentials of another superuser:

sentinel sentinel-user <username>

sentinel sentinel-pass <password>

Sentinel password-only authentication

Sentinels will do two things when configured this way:

1. Clients will be required to enter a password to send orders to Sentinels. This is self-evident, given how Redis handles such configuration directives in general.
2. Furthermore, this Sentinel instance will use the same password it used to access the local Sentinel to login into all other Sentinel instances it connects to.

This means that the required pass password must be the same across all Sentinel instances. This way, each Sentinel may communicate without having to create a password for each Sentinel to access the other Sentinels, which would be problematic.

Ensure your client library can send the AUTH command to Sentinel instances before using this setup.

Sentinel clients implementation

Unless the system is set to run a script that transparently redirects all requests to the new master instance, Sentinel requires explicit client support (virtual IP or other similar systems).

Sentinels and replicas auto-discovery

Sentinels maintain contact with other Sentinels to monitor each other's availability and exchange messages. Sentinel uses the Redis instance's Pub/Sub capabilities to discover other Sentinels monitoring the same masters and replicas, so you don't need to configure a list of other Sentinel addresses in every Sentinel instance you use to run.

This feature is activated by sending hello messages to the __sentinel :hello channel.

Sentinel will automatically discover the list of replicas attached to a master by querying Redis, so you don't have to set it up.

• Every two seconds, every Sentinel sends a message __sentinel :hello to every monitored master and replica Pub/Sub channel, declaring its presence with IP, port, and runid.
• Every Sentinel is subscribed to every master and replica's Pub/Sub channel __sentinel :hello, used to search for unknown sentinels. They are added to this master's sentinels when additional sentinels are discovered.
• The current configuration of the master is also included in the hello messages. If the receiving Sentinel has an older configuration for a specific master than the one received, it immediately changes to the new configuration.
• A Sentinel checks if there is already a sentinel with the same runid or address before adding it to a master (IP and port pair). In that situation, the old sentinels are deleted, and the new ones are installed.

Sentinel reconfiguration of instances outside the failover procedure

Sentinels will try to set the current configuration on monitored instances, even if no failover is in progress. Specifically:

• Replicas that claim to be masters will be configured as replicas to replicate with the current master (based on the existing configuration).
• Replicas associated with the incorrect master will be changed to duplicate with the correct master.

Sentinels must monitor the incorrect configuration for a longer period than the interval required to broadcast new configurations before they can reconfigure replicas.

This prohibits Sentinels with an out-of-date configuration (for example, those who have recently rejoined a partition) from attempting to change the replicas settings before getting an update.

Take note of how the semantics of constantly attempting to enforce the existing configuration makes the failover more partition-resistant:

• When failed masters become available again, they are reconfigured as replicas.
• When replicas that were partitioned away during a partition become reachable, they are reconfigured.

We can conclude that Sentinel is a system in which each process tries to impose the most recent logical configuration on the collection of monitored instances.

Replica selection and priority

The master is in ODOWN state, & the Sentinel got consent to failover from most of the Sentinel instances known. A suitable replica must be picked when a Sentinel instance is ready to undertake a failover.

The following information concerning replicas is evaluated during the replica selection process:

1. Time to disconnect from the master.
2. Prioritize the replica.
3. Processed replication offset
4. Run ID

A replica that has been detached from the master for more than ten times the configured master timeout, plus the time the master is unavailable from the perspective of the Sentinel performing the failover, is deemed unsuitable for failover and is skipped.

Frequently Asked Questions

What is Redis?

Redis is a key-value data storage and cache that is open-source.

What is the meaning of Redis?

Redis stands for REmote DIctionary Server.

What language is Redis written in?

Redis is a cache solution and session management system developed in ANSI C. It generates one-of-a-kind keys for storing data.

What is the usage of Redis?

Redis is a key-value store database that may be used as a NoSQL database or a memory cache store to boost performance when delivering data from system memory.

What are the ways to interact with Redis?

After the server has been installed, you can either use the Redis Client given by the Redis installation or open a command prompt and type the following command: redis-cli

Conclusion

In this article, we have learned about the High availability in Sentinel in redis. We hope that this blog will help you understand the concept of High availability in Sentinel in redis, and if you would like to learn more about it, check out our other blogs on redis, mongo-dB, database, and operational database.

Attempt our Online Mock Test Series on Coding Ninjas Studio now!

Happy Coding!