Redis Cluster allows a Redis installation to automatically partition its dataset across multiple nodes. Keys are partitioned using the concept of hash slots, of which there are 16384 in total, distributed across the cluster's nodes. For example, if the cluster consists of three nodes, the first node contains hash slots 0 to 5500, the second contains hash slots 5501 to 11000 and the third node contains hash slots 11001 to 16383. To determine a key's hash slot, we take the CRC16 of the key's modulo 16384.
One implication of the way keys are distributed across nodes is that operations that target multiple keys, such as MSET or MGET, issue more Redis commands than they would on a single Redis node setup.
To demonstrate, we will store the world's cities' populations on a Redis cluster. We will then retrieve the populations of all cities in Germany and sum them up to calculate the country's population. The supporting Python code is available on github and depends on the redis-py-cluster library.
First, let's setup a Redis cluster. There are several ways to run a Redis cluster, for example with Docker. Here, we download and build Redis from source, assuming a *nix system with make. After building Redis, we set up the create-cluster utility for a cluster consisting of six master and six slave Redis nodes.
$ ./setup-redis-cluster.sh
[...]
[OK] All 16384 slots covered.
We can now import city population data into the running cluster. One way to achieve this is by issuing one SET command per key. The keys' format is <country>_<city>, for example France_Paris.
$ ./measure.sh set.py OK OK OK OK OK OK real 0m6.439s user 0m2.978s sys0m0.863s 4460 4465 4439 4508 4314 4436
The time command is used as a crude measure of the execution time of the Python scripts that issue the Redis commands. The last six lines of the measure.sh script's output are total_commands_processed, as reported by each of the cluster's six master nodes. In this first attempt, we can see that key distribution is even across the cluster's nodes and that we've issued more than 4000 SET commands per node. This is consistent with our expectations of Redis Cluster's key hashing algorithm.
Insertion can be optimised by reducing the number of issued commands with the use of MSET. Calling the redis-py-cluster library's mset function with multiple keys doesn't have the desired effect. Its implementation iterates over the keys provided to it and issues a SET command for each. That's because a Redis Cluster node will only accept multi-key commands where all the keys hash to the same slot. To address this, Redis Cluster provides the concept of hash tags, according to which, if there is a substring between {} brackets in a key, only what is inside the string is hashed
. An example of this in our demo application would be {France}_Paris. By surrounding the country section of each key in curly braces, we are effectively storing them in the same slot, which means that we can store an entire group by calling MSET once. That is what the mset.py script does.
$ ./measure.sh mset.py OK OK OK OK OK OK real 0m0.844s user 0m0.563s sys0m0.097s 31 32 37 27 26 29
The demo application's mset.py script issues less Redis commands and executes faster than set.py.
Now suppose we want to determine Germany's population by fetching all of its cities' populations and summing them up. The demo application provides two scripts for doing this: get.py which issues one GET command per city and mget.py which uses one MGET command to fetch all of the cities' populations.
$ ./measure.sh get.py OK OK OK OK OK OK Population: 57857061 real 0m0.729s user 0m0.483s sys0m0.096s 295 257 262 262 278 272
$ ./measure.sh mget.py OK OK OK OK OK OK Population: 57857061 real 0m0.285s user 0m0.242s sys0m0.030s 5 2 3 3 3 3
Similar to when the cluster was populated, the MGET version is faster and issues less commands than its GET counterpart.
This technique can improve the performance of applications where groups of keys are frequently accessed together. However, it is worth noting that by grouping keys using hash tags, we are effectively overriding Redis Cluster's key hashing algorithm and potentially making keys distributed less evenly across nodes, which can cause certain nodes to use more memory than others.
This article was inspired by work we've done with my colleagues from the Supply team at loveholidays.com.
George Malamidis, 2021-04-24