Redis Basics

Why use Key-value storage system

According to the CAP theory in the distributed field, the ACID of traditional relational databases (such as Mysql) only satisfy consistency (Consistency) and availability (Availability), so it is difficult to do well in partition tolerance (Partition tolerance).

In addition, traditional relational databases also have great limitations in processing massive data in terms of Performance, Scalability, and Availability.

The Key-Value database pays more attention to the performance, distribution, and scalability support for massive data access. It does not require some features of traditional relational databases, such as Schema, transactions, complete SQL query support, etc. Therefore, the performance in a distributed environment is greatly improved compared to traditional relational databases.

In general, the Key-Value database sacrifices the Consistency of the CAP theory for better Partition tolerance and Availability

[scode type=“lblue”] CAP theory is called Brewer’s theorem. Among them, Consistency, Availability, and Tolerance to network Partitions can only satisfy two of them at the same time when implementing any system architecture, and cannot take into account all three.

Consistency: Equivalent to all nodes accessing the same latest copy of data
Availability: A non-error response can be obtained for every request - but there is no guarantee that the data obtained is the latest data)
Partition tolerance: In practical terms, partitioning is equivalent to the time limit requirement for communication. If the system cannot achieve data consistency within the time limit, it means that a partition has occurred and a choice must be made between C and A for the current operation.

ACID properties: In order to maintain database consistency, traditional relational databases follow ACID properties before and after transaction processing:

Atomicity: that is, indivisibility. Either none of the operations in the transaction are done, or all of them are done.
Consistency: Before and after a transaction is executed, the database must be in the correct state and satisfy the integrity constraints
Isolation: When multiple transactions are executed concurrently, the execution of one transaction should not affect the execution of other transactions
Durability: After the transaction is completed, the modification to the data is permanent and will not be lost even if the system fails.

[/scode]

RedisIntroduction

Redis is an open source log-type Key-Value database written in ANSI C language, supports network, can be memory-based and persistent, and provides APIs in multiple languages. As of March 15, 2010, development of Redis is hosted by VMware.

Data type of Redis

As a Key-value database, Redis also provides the mapping relationship between keys (Keys) and key values (Values). However, in addition to regular numeric values or strings, the key value of Redis can also be one of the following forms:

Lists
Sets
Sorted sets (ordered sets)
Hashes (hash table)

string data type

String is the simplest type, one key corresponds to one value. The string type is binary safe. This means that Redis string can contain any data, such as jpg images or serialized objects. From the internal implementation point of view, string can be regarded as a byte array. The maximum limit is 1G bytes. The following is the definition of string type:

struct sdshdr {
    long len;
    long free;
    char buf[];
};

len is the length of the buf array. free is the number of available bytes remaining in the array. From this, we can understand why the string type is binary safe, because it is essentially a byte array and can of course contain any data. buf is a char array used to store actual string content. In fact, char and byte in C# are equivalent, both are one byte.

In addition, the string type can be processed as int by some commands. For example, incr and other commands, if only the string type is used, Redis can be regarded as memcached with persistence features. Of course, Redis still has many more operations on string types than memcached. Here is a list of common methods (please consult the manual for other less commonly used methods):

####Commonly used operation commands for string data type

set

Set the value corresponding to key to a value of type string.

setex

Set the value corresponding to key to a string type value, and specify the validity period corresponding to this key value.

get

Get the string value corresponding to key, and return nil if key does not exist.

incr

Performs an addition operation on the value of key and returns the new value. Note that if incr is a value that is not int, an error will be returned. If incr is a non-existent key, set the key to 1.

incrby

Similar to incr, it specifies the incremental step size. When the key does not exist, the key will be set and the original value is considered to be 0.

decr

The value of the key is subtracted. If the decr key does not exist, set the key to -1.

decrby

Same as incrby, just subtracted

strlen

Get the length of the value value of the specified key.

setnx

Set the value corresponding to key to a value of type string. If key already exists, return 0, nx means not exist.

getset

Sets the value of key and returns the old value of key.

hset

Almost the most commonly used ones are the above 11. If you are developing, usually there is a third-party package closed method to achieve it. Just understand the principle.

lists types and operations

List is a linked list structure, its main functions are push, pop, getting all values in a range, etc. During the operation, key is understood as the name of the linked list.

The list type of Redis is actually a doubly linked list in which each sub-element is of string type. We can add and delete elements from the head or tail of the linked list through push and pop operations. This allows the list to be used as both a stack and a queue.

Sorted Set data type

The usage scenario of Redis sorted set is similar to that of set. The difference is that set is not automatically ordered, while sorted set can sort members by providing an additional priority (score) parameter by the user, and it is insertion ordered, that is, automatically sorted. When you need an ordered and non-duplicate set list, you can choose the sorted set data structure. For example, Twitter’s public timeline can store the publication time as the score, so that it will be automatically sorted by time when retrieved.

Redis sorted set internally uses HashMap and skip list (SkipList) to ensure the storage and ordering of data. HashMap stores the mapping from members to score, while the skip list stores all members. The sorting is based on the score stored in HashMap. Using the structure of the skip list can achieve higher search efficiency and is relatively simple to implement.

Redis Management common commands

Key value command

keys

Returns all keys that satisfy the given pattern

127.0.0.1:6379> keys *
1) "mylist"
2) "bar"

[scode type=“yellow”] Be careful when executing this command in a production environment, because the production environment has a lot of caches and may consume more system resources if executed. [/scode]

exists

Confirm whether a key exists

del

Delete a key

expire

Set the expiration time of a key (unit: seconds)

move

Transfer keys in the current database to other databases

rename

Rename key

type

Return value type

Server commands

ping

Test whether the connection is alive

127.0.0.1:6379> ping
PONG

Returning pong means the connection is normal, others means the connection is abnormal.

select

Select a database. Redis database numbers range from 0 to 15. We can choose any database to access data.

quit

Exit the connection.

info

Get server information and statistics.

flushdb

Delete all keys in the currently selected database.

flushall

Delete all keys in all databases.

config get

Get server configuration information.

127.0.0.1:6379> config get dir
1) "dir"
2) "/usr/local/var/db/redis"

dbsize

Returns the number of keys in the current database.

For usage of other commands, please refer to the manual.

Advanced practical features of Redis

Security

The client connection service can set a password to prevent unauthorized connections. [scode type=“yellow”] Warning: Because Redis is quite fast, on a good server, an external user can make 150K password attempts per second, which means you need to specify very, very strong passwords to prevent brute force attacks. The best approach is to put it together with the application on the intranet and behind the firewall. [/scode]

Master-slave replication

Redis Master-slave replication is very simple to configure and use. Master-slave replication allows multiple slave servers to own and The same database copy as the master server.

Redis Characteristics of master-slave replication

The master can have multiple slaves
Multiple slaves can connect to the same master or other slaves
Master-slave replication will not block the master. When synchronizing data, the master can continue to process client requests.
Improve system scalability

After the slave is configured, the slave establishes a connection with the master and then sends the sync command. Whether it is the first connection or reconnection, the master will start a background process to save the database snapshot to a file. At the same time, the master main process will start to collect new write commands and cache them.

After the background process completes writing the file, the master sends the file to the slave. The slave saves the file to the hard disk and loads it into the memory. Then the master forwards the cached command to the slave. Subsequently, the master sends the received write command to the slave. If the master receives synchronous connection commands from multiple slaves at the same time, the master will only start a process to write the database mirror, and then send it to all slaves.

How to configure

Configuring the slave server is very simple. You only need to add the following configuration to the slave configuration file.

slaveof 192.168.1.1 6379 #指定 master 的 ip 和端口

How to determine which one comes from which one?

Execute the info command:

redis 127.0.0.1:6378> info .
.
.
role:slave
master_host:localhost master_port:6379 master_link_status:up master_last_io_seconds_ago:10 master_sync_in_progress:0 db0:keys=1,expires=0

There is a role identifier inside to determine whether it is the master library or the slave library. In this example, it is a slave library. There is also a master_link_status used to indicate whether the master-slave is asynchronous. If this value = up, it means the synchronization is normal; if this value = down, it means the synchronization is asynchronous;

Transaction Control

This is not recommended. Please use a relational database transaction instead, or use a program to implement distributed transactions.

Data persistence

Typically, Redis stores data in memory, or is configured to use virtual memory. In other words, Redis needs to frequently synchronize data in memory to disk to ensure persistence. Redis supports two persistence methods

RDB persistence: Use snapshotting to continuously write data in memory to disk;
AOF persistence: Each write operation received by the server will be recorded. When the service starts, these recorded operations will be executed one by one to reconstruct the original data. The format of write operation command records is consistent with the Redis protocol and is saved in an appending manner (somewhat similar to Mysql’s binlog).

snapshotting method

Snapshots are the default persistence method. This method is to write the data in the memory into a binary file in the form of a snapshot. The default file name is dump.rdb.

Snapshot persistence can be automatically done through configuration settings. We can configure Redis to automatically take a snapshot if more than m keys are modified within n seconds. The following is the default snapshot saving configuration.

save 900 1 #900 秒内如果超过 1 个 key 被修改，则发起快照保存
save 300 10 #300 秒内容如超过 10 个 key 被修改，则发起快照保存
# 以此类推

Since Redis uses one main thread to handle all client requests, this method will block all client requests. So its use is not recommended. Another point to note is that each snapshot persistence writes the memory data to the disk completely once, rather than incrementally synchronizing only the changed data. If the amount of data is large and there are many write operations, it will inevitably cause a large number of disk IO operations, which may seriously affect performance.

aof method

In addition, since the snapshot method is done at a certain interval, if Redis accidentally goes down, all modifications after the last snapshot will be lost. If the application requires that no modifications can be lost, the aof persistence method can be used.

aof has better persistence than the snapshot method because when using the aof persistence method, Redis will append every received write command to the file through the write function (the default is appendonly.aof).

When Redis is restarted, the contents of the entire database will be rebuilt in memory by re-executing the write commands saved in the file.

Of course, since the OS will cache the modifications made by write in the kernel, it may not be written to the disk immediately. In this way, aof persistence may still lose some modifications.

However, we can tell Redis through the configuration file when we want to force os to write to the disk through the fsync function. There are three ways as follows (default: fsync once per second)

appendonly yes  // 启用 aof 持久化方式
# appendfsync always // 收到写命令就立即写入磁盘，最慢，但是保证完全的持久化
appendfsync everysec // 每秒钟写入磁盘一次，在性能和持久化方面做了很好的折中
# appendfsync no // 完全依赖os, 性能最好，持久化没保证

This function requires specific application scenarios.

Pipeline sends requests in batches

It is only needed when the network is not very good, so avoid it from the architecture.

Virtual memory usage

It’s better to add money to buy memory!