Detailed explanation of Redis related commands and their principles: Redis basic operations, data structures and applications

How to learn redis?

  • Learn what redis is
  • How to use redis, how to operate the data structure in redis
  • Typical applications and operations
  • Read the redis source code

1. Redis

Redis is the abbreviation of Remote Dictionary Service; it is also a remote dictionary service;
Redis is an in-memory database, KV database, data structure database;
Redis is widely used, such as Twitter, Blizzard Entertainment, Github, Stack Overflow, Tencent, Alibaba, Beijing
East, Huawei, Sina Weibo, etc., are also used by many small and medium-sized companies;
Redis command to view: http://redis.cn/commands.html

2. Application

  • Record the number of likes, comments and clicks (hash) in the circle of friends
  • Record the list of friends circle (sorting), which is convenient to quickly display the circle of friends (list)
  • Record the title, abstract, author and cover of the article for list page display (hash)
  • Record the list of like user IDs and comment IDs in the circle of friends for display and deduplication count (zset)
  • Cache hot data to reduce database pressure (hash)
  • If the Moments Talk ID is an integer id, you can use redis to assign the Moments Talk ID (counter) (string)
  • Recording friendship (set) through the intersection and subtraction of sets (sets)
  • In the game business, the record of each round is stored (list)

Three, installation and compilation

git clone https://gitee.com/mirrors/redis.git -b 6.2
cd redis
make
make test
make install
# Installed by default in /usr/local/bin
# redis-server is the server program
# redis-cli is the client program

start up

mkdir redis-data
# Copy redis.conf in the redis folder to redis-data
# Modify redis.conf
# requirepass change password 123456
# daemonize yes
cd redis-data
redis-server redis.conf
# Access redis-server through redis-cli
redis-cli -h 127.0.0.1 -a 123456

If the current host uses

redis-server redis.conf
redis-cli
auth "123456"

Fourth, know redis

Redis is an internal database, kv database, data structure database
There are string,hash,list,set,zset

The server sends a command to redis to operate the data structure in redis, and redis responds to the operation result

redis has 16 databases by default
redis can only be used in a single thread, and only one database can be used at the same time, which is organized in a dictionary, while mysql uses b+ tree organization

data structures in redis

string is a safe binary string (safety: string in redis is a safe string and will not be separated by '\0')
Hash: key-value pair
Double-ended queue (linked list) list: ordered (insertion ordered)
Unordered collection set: does not pay attention to the order, the values ​​in it are all unique
Ordered set zset: focus on order, the value in it is unique

5. Basic use of redis

Open the redis server

Open the redis server, redis.conf is the configuration file of the server

redis-server redis.conf

Check if the server is open

ps aux|grep redis-server

The redis client connects to the server

Do not add anything after it, which means logging in to the local redis-server

redis-cli

After the client enters
it looks like this

It can only be used after password authentication. Note that the password is in the form of a string, and double quotation marks should be added.

auth "yourpassword"  

key-value

Set the value of key
where key is "hello" and value is "world"
(is not repeatable)

set hello world

Get value by key

get hello

hset

set a hash
key=markinfo points to a key-value pair, filed=age, and the corresponding hash value.
Can be understood as a secondary dictionary
(is not repeatable)

hset markinfo age 30

Get the value corresponding to age in the value markinfo

hget markinfo age

list

(content can be repeated)
Insert the three elements "dog", "cat", "dog" into the linked list liststr

lpush liststr dog cat dog

Take data from liststr, begin=0 represents the starting position end=-1 represents the end

lrange liststr 0 -1

pop out the first value of liststr

lpop liststr

view all key s

keys *

Like redis, mysql is a strictly ordered request-response mode.
And mongo does not necessarily respond in the order requested

Basic steps to use redis

The first step is to connect
The second step is to enter the password for auth, if it is not set, it is not required
The third step is to select the database. If this step is not performed, the default is database 0.

Six, data structure: string

Character array, the string is a dynamic string raw, when the length of the string is less than 1M, double expansion; more than 1M, only more expansion each time
1M; the maximum length of the string is 512M;
Note: The redis string is a binary safe string; it can store binary data such as pictures, binary protocols, etc.;

basic command

# Set the value of key
SET key val
# Get the value of key
GET key
# Perform an atomic increment operation
INCR key
# Perform an atomic addition of an integer
INCRBY key increment
# Perform an atomic subtraction operation
DECR key
# perform atomic subtraction of an integer
DECRBY key decrement
# If the key does not exist, this case is equivalent to the SET command.  When the key exists, do nothing
SETNX key value
# delete key val key-value pair
DEL key
# Sets or clears the bit value of the key's value (string) at offset.
SETBIT key offset value
# Returns the bit value of the string corresponding to the key at offset
GETBIT key offset
# The number of bit s in which the statistics string is set to 1.
BITCOUNT key

The difference between setnx and set:
setnx is set not exist, for existing key s, it will not be processed
And set is a setting whether it exists or not, and it will be overwritten if it exists.

storage structure

If the string length is less than or equal to 20 and can be converted to an integer, use int to store;
If the string length is less than or equal to 44, use embstr to store;
If the string length is greater than 44, use raw storage;

All key s in redis are string s


Different types are used to save memory
You can view the stored data type through object encoding [key]

string can be used as a bitmap

Also available as a bitmap
Set index 9 to 1. You can see that index 9 is 1, and index 8 is 0 (because it is not set)

You can also view the number of 1s in the bitmap

application

object storage

SET role:10001 '{["name"]:"mark",["sex"]:"male",["age"]:30}'
# Rarely modified, when the object attribute field rarely changes, you can write this way, otherwise it is recommended to use hash
GET role:10001

The number after the colon is used to uniquely identify that key
Open the redis client and you can see that role is a folder that contains role:10001

Therefore, you can continuously define the key by adding a colon, which can be understood as a folder (tree structure)
Such as

role:10001:recharge
# or multi-level colons
role:10001:activity:10001

accumulator

# Count the number of readings, add 1 to the total
incr reads
# Add 100 cumulatively
incrby reads 100

Using incr key must be guaranteed to convert string to int, otherwise it will fail

Distributed lock

# lock   
setnx lock 1 # Can only be set if it does not exist Define locking behavior Occupy lock
# release lock
del lock
if(get(lock)==uuid)
	del(lock);
# 1. Exclusive function 2. Lock behavior definition 3. Release behavior definition

uuid represents the unique identifier of the client, nx represents not exist (the set can only be successful if it does not exist), and ex 30 represents (expire expires after 30 seconds)

set lock uuid nx ex 30

Mutex is a kind of fair lock. A acquires the lock, operates the critical resource, and after releasing the lock, acquires the lock in the requested order.
The spin lock is an unfair lock that acquires the lock through continuous attempts (there is no strict order, whoever requests the lock will acquire the lock).
A distributed lock is an unfair lock.

Seven, data structure: list

Doubly linked list implementation, the time complexity of the head and tail operations (deletion and addition) of the list is O(1); the time complexity of finding intermediate elements is
O(n) ;
The basis for whether the data in the list is compressed:

  1. Element length is less than 48, no compression;
  2. The length difference between the elements before and after compression does not exceed 8, and no compression is performed;

basic command

# Enqueue one or more elements from the left side of the queue
LPUSH key value [value ...]
# Pop an element from the left side of the queue
LPOP key
# Enqueue one or more elements from the right side of the queue
RPUSH key value [value ...]
# Pop an element from the right side of the queue
RPOP key
# Returns the elements 0, 1 2 from the queue between start and end
LRANGE key start end
# Removes the first count occurrences of the element whose value is value from the list stored at key
LREM key count value
# It is a blocking version of RPOP, as this command blocks the connection if no elements can be popped from the given list
BRPOP key timeout # timeout + delay queue

storage structure

/* Minimum ziplist size in bytes for attempting compression. */
#define MIN_COMPRESS_BYTES 48
/* quicklistNode is a 32 byte struct describing a ziplist for a quicklist.
* We use bit fields keep the quicklistNode at 32 bytes.
* count: 16 bits, max 65536 (max zl bytes is 65k, so max count actually <
32k).
* encoding: 2 bits, RAW=1, LZF=2.
* container: 2 bits, NONE=1, ZIPLIST=2.
* recompress: 1 bit, bool, true if node is temporary decompressed for
usage.
* attempted_compress: 1 bit, boolean, used for verifying during testing.
* extra: 10 bits, free for future use; pads out the remainder of 32 bits */
typedef struct quicklistNode {
struct quicklistNode *prev;
struct quicklistNode *next;
unsigned char *zl;
unsigned int sz; /* ziplist size in bytes */
unsigned int count : 16; /* count of items in ziplist */
unsigned int encoding : 2; /* RAW==1 or LZF==2 */
unsigned int container : 2; /* NONE==1 or ZIPLIST==2 */
unsigned int recompress : 1; /* was this node previous compressed? */
unsigned int attempted_compress : 1; /* node can't compress; too small
*/
unsigned int extra : 10; /* more bits to steal for future usage */
} quicklistNode;
typedef struct quicklist {
quicklistNode *head;
quicklistNode *tail;
unsigned long count; /* total count of all entries in all
ziplists */
unsigned long len; /* number of quicklistNodes */
int fill : QL_FILL_BITS; /* fill factor for individual
nodes */
unsigned int compress : QL_COMP_BITS; /* depth of end nodes not to
compress;0=off */
unsigned int bookmark_count: QL_BM_BITS;
quicklistBookmark bookmarks[];
} quicklist;

application

stack (first in, last out FILO)

LPUSH + LPOP
# or
RPUSH + RPOP

Queue (First In First Out FIFO)

LPUSH + RPOP
# or
RPUSH + LPOP

blocking queue

LPUSH + BRPOP
# or
RPUSH + BLPOP

Redis Lrem Command

Asynchronous message queue

Operations are the same as queues, but between different systems

The web sends data to redis, and several servers obtain data by blocking brpop or blpop

Get fixed window records

is to maintain a fixed window size of data

method:
1. Fixed window current limiting
2. Truncating current limiting

Redis Ltrim Commands
Only the most recent 50 pieces of data are kept by ltrim to achieve the effect of current limiting

Eight, data structure: hash

Hash table, which is a data structure in many high-level languages; c++ unordered_map is quickly indexed by key
value;

basic command

# Get the value corresponding to the field in the hash corresponding to the key
HGET key field
# Set the value corresponding to the field in the hash corresponding to the key
HSET key field value
# Set multiple hash key-value pairs
HMSET key field1 value1 field2 value2 ... fieldn valuen
# Get the value of multiple field s
HMGET key field1 field2 ... fieldn
# Add an integer value to the value corresponding to the field in the hash corresponding to the key
HINCRBY key field increment
# Get the number of key-value pairs in the hash corresponding to the key
HLEN key
# Delete the key-value pair of hash corresponding to key, the key is field
HDEL key field

storage structure

If the number of nodes is greater than 512 (hash-max-ziplist-entries) or the length of all strings is greater than 64 (hash-max-ziplistvalue), use dict to implement;
If the number of nodes is less than or equal to 512 and there is a string whose length is less than 64, use ziplist to implement;

application

storage object

hmset hash:10001 name mark age 18 sex male
# compare with string
set hash:10001 '{["name"]:"mark",["sex"]:"male",["age"]:18}'
# Suppose now that the age of the modified mark is 19 years old
# hash: 
hset hash:10001 age 19
# string:
get role:10001
# Decrypt the obtained string by calling json, take out the field, and modify the age value
# Then call json encryption
set role:10001 '{["name"]:"mark",["sex"]:"male",["age"]:19}'

shopping cart

# use user id as key
# Product id as field
# Item quantity as value
# Note: These items are displayed in the order we added them;
# Adding goods:
hset MyCart:10001 40001 1
lpush MyItem:10001 40001
# Increase the number of:
hincrby MyCart:10001 40001 1
hincrby MyCart:10001 40001 -1 // reduce the number by 1
# Show all item quantities:
hlen MyCart:10001
# To delete an item:
hdel MyCart:10001 40001
lrem MyItem:10001 1 40001
# Get all items:
lrange MyItem:10001
# 40001 40002 40003
hget MyCart:10001 40001
hget MyCart:10001 40002
hget MyCart:10001 40003

Nine, data structure: set

Collection; used to store unique fields, does not require ordering;
Storage does not need ordering, operation (ordering when intersecting and subtracting sets)

basic command

# Adds one or more specified member elements to the set's key
SADD key member [member ...]
# Count the number of elements in a collection
SCARD key
# SMEMBERS key
SMEMBERS key
# Returns whether member member is a member of the stored collection key
SISMEMBER key member
# Randomly returns one or more elements in the key set, without deleting these elements
SRANDMEMBER key [count]
# Remove and return one or more random elements from the collection stored at key
SPOP key [count]
# Returns the elements of the difference between a set and the given set
SDIFF key [key ...]
# Returns the intersection of all members of the specified set
SINTER key [key ...]
# Returns all members of the union of the given collections
SUNION key [key ...]

Since the set is stored unordered, the result of spop should be a random value

storage structure

If the elements are integers and the number of nodes is less than or equal to 512 (set-max-intset-entries), use integer array storage;
If one of the elements is not an integer or the number of nodes is greater than 512, it is stored in a dictionary;

application

lottery

# Add lottery users
sadd Award:1 10001 10002 10003 10004 10005 10006
sadd Award:1 10009
# View all raffle users
smembers Award:1
# Draw multiple lucky users
srandmember Award:1 10
# What should I do if I draw 1 first prize, 2 second prizes, and 3 third prizes?

Common concern

sadd follow:A mark king darren mole vico
sadd follow:C mark king darren
sinter follow:A follow:C

Refer a friend

sadd follow:A mark king darren mole vico
sadd follow:C mark king darren
# People C may know:
sdiff follow:A follow:C

Ten, data structure: zset

Ordered collection; used to implement leaderboards; it is an ordered unique;

basic command

# Add to the key ed sorted set (sorted set)
ZADD key [NX|XX] [CH] [INCR] score member [score member ...]
# Deletes the key-value pair of member from an ordered set whose key is key
ZREM key member [member ...]
# Returns the score value of member member in the sorted set key
ZSCORE key member
# Add an increment to the score value of the member member of the sorted set key
ZINCRBY key increment member
# Returns the number of elements in the sorted set for key
ZCARD key
# Returns the rank of member member in the sorted set key
ZRANK key member
# Returns the specified range of elements stored in the sorted set key order by id limit 1,100
ZRANGE key start stop [WITHSCORES]
# Returns the members in the specified range in the ordered set key (reverse order)
ZREVRANGE key start stop [WITHSCORES]

byscore is sorted according to the score, withscores also prints the score

Sort from largest to smallest

storage structure

If the number of nodes is greater than 128 or there is a string whose length is greater than 64, a skiplist is used;
If the number of nodes is less than or equal to 128 (zset-max-ziplist-entries) and the length of all strings is less than or equal to 64 (zset-maxziplist-value), use ziplist storage;
Save space when data is small; O(n)
When the number is large, the access performance; O(1) o(logn)

application

Baidu Hot List

# Click for news:
zincrby hot:20210601 1 10001
zincrby hot:20210601 1 10002
zincrby hot:20210601 1 10003
zincrby hot:20210601 1 10004
zincrby hot:20210601 1 10005
zincrby hot:20210601 1 10006
zincrby hot:20210601 1 10007
zincrby hot:20210601 1 10008
zincrby hot:20210601 1 10009
zincrby hot:20210601 1 10010
# Get leaderboards:
zrevrange hot:20210601 0 9 withscores

delay queue

Serialize the message into a string as a member of zset; the expiration processing time of this message is used as score, and then use
Multiple threads poll zset for expired tasks for processing

def delay(msg):
	msg.id = str(uuid.uuid4()) #Guarantee member unique
	value = json.dumps(msg)
	retry_ts = time.time() + 5 # Retry after 5s
	redis.zadd("delay-queue", retry_ts, value)
# Use connection pool
def loop():
	while True:
		values = redis.zrangebyscore("delay-queue", 0, time.time(), start=0,
num=1)
	if not values:
		time.sleep(1)
		continue
	value = values[0]
	success = redis.zrem("delay-queue", value)
	if success:
		msg = json.loads(value)
		handle_msg(msg)
	# Disadvantages: loop is a multi-threaded competition, both threads get data from zrangebyscore, but zrem one succeeds and the other fails,
	# Optimization: To avoid redundant operations, these two commands can be executed atomically using a lua script
	# Solution: funnel flow restriction

Distributed timer

The producer hash es scheduled tasks to different redis entities, and assigns a dispatcher process to each redis entity.
It is used to regularly obtain timeout events in redis and publish them to different consumers;

Time window current limit

The system restricts that a user's behavior can only occur N times within a specified time range (dynamically);

# A certain behavior of the specified user user_id is only allowed to do more times max_count within a certain period of time
local function is_action_allowed(red, userid, action, period, max_count)
	local key = tab_concat({"hist", userid, action}, ":")
	local now = zv.time()
	red:init_pipeline()
	-- record behavior
	red:zadd(key, now, now)
	-- Remove the behavior records before the time window, and the rest are records within the time window
	red:zremrangebyscore(key, 0, now - period *100)
	-- Get the number of actions within a time window
	red:zcard(key)
	-- Set the expiration time to prevent cold users from continuously occupying memory Length of time window+1 Second
	red:expire(key, period + 1)
	local res = red:commit_pipeline()
	return res[3] <= max_count
	end
# Maintain a time window, clear all records outside the window, and keep only the records within the window;
# Disadvantages: data in all time windows are recorded, if this amount is large, it is not suitable for such current limiting; funnel current limiting
# Note: If the key + expire operation can also be implemented, but the implementation is fuse, and the maintenance time window is the function of current limiting;

Tags: Database Redis data structure

Posted by cryp7 on Thu, 05 May 2022 16:53:18 +0300