博客 / 詳情

返回

Redis 集羣搭建

集羣簡介

Redis 集羣是一個可以在多個 Redis 節點之間進行數據共享的設施（installation）。

Redis 集羣不支持那些需要同時處理多個鍵的 Redis 命令， 因為執行這些命令需要在多個 Redis 節點之間移動數據， 並且在高負載的情況下， 這些命令將降低 Redis 集羣的性能，並導致不可預測的行為。

Redis 集羣通過分區（partition）來提供一定程度的可用性（availability）：即使集羣中有一部分節點失效或者無法進行通訊，集羣也可以繼續處理命令請求。

Redis 集羣提供了以下兩個好處：

  • 將數據自動切分（split）到多個節點的能力
  • 當集羣中的一部分節點失效或者無法進行通訊時，仍然可以繼續處理命令請求的能力

集羣説明

集羣數據共享

Redis 集羣使用數據分片（sharding）而非一致性哈希（consistency hashing）來實現：一個 Redis 集羣包含 16384 個哈希槽（hash slot），數據庫中的每個鍵都屬於這 16384 個哈希槽的其中一個，集羣使用公式 CRC16(key) % 16384 來計算鍵 key 屬於哪個槽，其中 CRC16(key) 語句用於計算鍵 key 的 CRC16 校驗和。

集羣中的每個節點負責處理一部分哈希槽

一個集羣可以有三個哈希槽，其中：

  • 節點 A 負責處理 0 號至 5500 號哈希槽
  • 節點 B 負責處理 5501 號至 11000 號哈希槽
  • 節點 C 負責處理 11001 號至 16384 號哈希槽

集羣中的主從複製

為了使得集羣在一部分節點下線或者無法與集羣的大多數（majority）節點進行通訊的情況下，仍然可以正常運作，Redis 集羣對節點使用了主從複製功能：集羣中的每個節點都有 1 個至 N 個複製品（replica），其中一個複製品為主節點（master），而其餘的 N-1 個複製品為從節點（slave）

集羣 TCP 端口

每個 Redis Cluster 節點都需要 2 個 TCP 連接打開，正常的 Redis TCP 端口被服務於客户端，例如：6379，加上通過增加 10000 到數據端口獲取的端口，例如：16379。

第二個高端口被用來 Redis Cluster Bus，也就是使用二進制協議的一個節點到節點的通信信道。Redis Cluster Bus 被節點用於故障檢測，配置更新，故障轉移授權等等。客户端不應該嘗試與 Redis Cluster Bus 端口進行通信，而是始終使用正常的 Redis 命令端口，然而要確保在防火牆中打開兩個端口，否則 Redis Cluster 節點將無法通信。

命令端口和 Redis Cluster Bus 端口偏移是固定的，始終為 10000。

注意，為了使 Redis Cluster 正常工作，你需要為每個節點：

  • 用於客户端進行通信的普通客户端通信端口（通常使用 6379）對所有需要到達集羣的客户端，以及所有其它集羣節點（使用客户端端口進行密鑰遷移）都是開放的
  • Redis Cluster Bus 端口（客户端端口 +10000）必須可從所有其它集羣節點訪問

如果不打開兩個 TCP 端口，集羣將無法按預期工作

Redis Cluster Bus 使用不同的二進制協議進行節佔到節點的數據交換，這更適合於使用少量的帶寬和處理時間交換節點之間的信息

安裝依賴

$ sudo gpg --keyserver hkp://keys.gnupg.net --recv-keys 409B6B1796C275462A1703113804BB82D39DC0E3
$ sudo curl -sSL https://get.rvm.io | bash -s stable
$ sudo source ~/.rvm/scripts/rvm
$ sudo echo "ruby_url=https://cache.ruby-china.org/pub/ruby" > ~/.rvm/user/db
$ rvm -v
$ sudo rvm install 2.4.1
$ sudo gem update
$ sudo gem install redis

集羣配置

安裝 Redis

$ cd ~
$ wget http://download.redis.io/releases/redis-4.0.9.tar.gz
$ tar xzf redis-4.0.9.tar.gz
$ cd redis-4.0.9
$ sudo make

創建配置目錄

$ sudo mkdir -pv /usr/local/redis-cluster/{bin,etc,log,run,var}

創建配置文件

$ sudo cp ~/redis-4.0.9/redis.conf /usr/local/redis-cluster/etc/redis-7001.conf

複製命令文件

$ cd ~/redis-4.0.9/src/
$ sudo cp mkreleasehdr.sh redis-benchmark redis-check-aof redis-check-rdb redis-sentinel redis-cli redis-server /usr/local/redis-cluster/bin

修改配置文件

$ sudo vim /usr/local/redis-cluster/etc/redis-7001.conf
bind 192.168.33.110
port 7001
daemonize yes

pidfile /user/local/redis-cluster/run/7001.pid
logfile /usr/local/redis-cluster/log/7001.log
dbfilename dump-7001.rdb

dir /usr/local/redis-cluster/var

appendonly yes
appendfilename "appendonly-7001.aof"

cluster-enabled yes
cluster-config-file nodes-7001.conf
cluster-node-timeout 15000

複製配置文件

將複製後的文件中的 7001 換成相應的名稱 
$ sudo cp /usr/local/redis-cluster/etc/redis-7001.conf /usr/local/redis-cluster/etc/redis-7002.conf
$ sudo cp /usr/local/redis-cluster/etc/redis-7001.conf /usr/local/redis-cluster/etc/redis-7003.conf
$ sudo cp /usr/local/redis-cluster/etc/redis-7001.conf /usr/local/redis-cluster/etc/redis-7004.conf
$ sudo cp /usr/local/redis-cluster/etc/redis-7001.conf /usr/local/redis-cluster/etc/redis-7005.conf
$ sudo cp /usr/local/redis-cluster/etc/redis-7001.conf /usr/local/redis-cluster/etc/redis-7006.conf

集羣部署

啓動節點

// 啓動配置的所有節點
$ sudo /usr/local/redis-cluster/bin/redis-cli /usr/local/redis-cluster/etc/redis-7001.conf
$ sudo /usr/local/redis-cluster/bin/redis-cli /usr/local/redis-cluster/etc/redis-7002.conf
$ sudo /usr/local/redis-cluster/bin/redis-cli /usr/local/redis-cluster/etc/redis-7003.conf
$ sudo /usr/local/redis-cluster/bin/redis-cli /usr/local/redis-cluster/etc/redis-7004.conf
$ sudo /usr/local/redis-cluster/bin/redis-cli /usr/local/redis-cluster/etc/redis-7005.conf
$ sudo /usr/local/redis-cluster/bin/redis-cli /usr/local/redis-cluster/etc/redis-7006.conf

檢查服務

$ ps -ef | grep redis
root     20077     1  0 May18 ?        00:01:04 /usr/local/bin/redis-server 192.168.33.110:7001 [cluster]
root     20079     1  0 May18 ?        00:01:03 /usr/local/bin/redis-server 192.168.33.110:7002 [cluster]
root     20081     1  0 May18 ?        00:01:03 /usr/local/bin/redis-server 192.168.33.110:7003 [cluster]
root     20089     1  0 May18 ?        00:01:03 /usr/local/bin/redis-server 192.168.33.110:7004 [cluster]
root     20091     1  0 May18 ?        00:01:03 /usr/local/bin/redis-server 192.168.33.110:7005 [cluster]
root     20096     1  0 May18 ?        00:01:03 /usr/local/bin/redis-server 192.168.33.110:7006 [cluster]

停止節點

$ sudo /usr/local/redis-cluster/bin/redis-cli -c -h 192.168.33.110 -p 7001 shutdown

節點加入集羣

$ sudo cp ~/redis-4.0.9/src/redis-trib.rb /usr/local/redis-cluster/bin/redis-trib
$ sudo chmod a+x /usr/local/redis-cluster/bin/redis-trib
$ sudo /usr/local/redis-cluster/bin/redis-trib create --replicas 1 \
> 192.168.33.110:7001 \
> 192.168.33.110:7002 \
> 192.168.33.110:7003 \
> 192.168.33.110:7004 \
> 192.168.33.110:7005 \
> 192.168.33.110:7006 \
>>> Creating cluster
>>> Performing hash slots allocation on 6 nodes...
Using 3 masters:
127.0.0.1:7001
127.0.0.1:7002
127.0.0.1:7003
Adding replica 127.0.0.1:7005 to 127.0.0.1:7001
Adding replica 127.0.0.1:7006 to 127.0.0.1:7002
Adding replica 127.0.0.1:7004 to 127.0.0.1:7003
>>> Trying to optimize slaves allocation for anti-affinity
[WARNING] Some slaves are in the same host as their master
M: 4ed7d8f54311fb883431c1c5a4715c1ae59d5215 127.0.0.1:7001
   slots:0-5460 (5461 slots) master
M: 31030e4c35659d5de1146941a2d1f2c1524b53e9 127.0.0.1:7002
   slots:5461-10922 (5462 slots) master
M: 573752b8b0747bc2c7bb3e4a1edb0e467cfdef0e 127.0.0.1:7003
   slots:10923-16383 (5461 slots) master
S: 11f577639b081c1514f28a74764c9330d819dcb8 127.0.0.1:7004
   replicates 573752b8b0747bc2c7bb3e4a1edb0e467cfdef0e
S: ff6d7ca3ea29036385c89a5dc93f807f91307871 127.0.0.1:7005
   replicates 4ed7d8f54311fb883431c1c5a4715c1ae59d5215
S: a6f79f4d1de8d7bef4911e934b802829ed29a2d3 127.0.0.1:7006
   replicates 31030e4c35659d5de1146941a2d1f2c1524b53e9
Can I set the above configuration? (type 'yes' to accept): yes
>>> Nodes configuration updated
>>> Assign a different config epoch to each node
>>> Sending CLUSTER MEET messages to join the cluster
Waiting for the cluster to join..
>>> Performing Cluster Check (using node 127.0.0.1:7001)
M: 4ed7d8f54311fb883431c1c5a4715c1ae59d5215 127.0.0.1:7001
   slots:0-5460 (5461 slots) master
   1 additional replica(s)
S: 11f577639b081c1514f28a74764c9330d819dcb8 127.0.0.1:7004
   slots: (0 slots) slave
   replicates 573752b8b0747bc2c7bb3e4a1edb0e467cfdef0e
S: a6f79f4d1de8d7bef4911e934b802829ed29a2d3 127.0.0.1:7006
   slots: (0 slots) slave
   replicates 31030e4c35659d5de1146941a2d1f2c1524b53e9
S: ff6d7ca3ea29036385c89a5dc93f807f91307871 127.0.0.1:7005
   slots: (0 slots) slave
   replicates 4ed7d8f54311fb883431c1c5a4715c1ae59d5215
M: 31030e4c35659d5de1146941a2d1f2c1524b53e9 127.0.0.1:7002
   slots:5461-10922 (5462 slots) master
   1 additional replica(s)
M: 573752b8b0747bc2c7bb3e4a1edb0e467cfdef0e 127.0.0.1:7003
   slots:10923-16383 (5461 slots) master
   1 additional replica(s)
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.

可以看到，創建集羣這個過程比較繁瑣，可以創建腳本來完成這些操作

  • 啓動腳本

    $ sudo vim /usr/local/redis-cluster/bin/start.sh
#!/usr/bin/env bash

RedisServerPath=/usr/local/redis-cluster/bin
RedisConfPath=/usr/local/redis-cluster/etc

for i in 1 2 3 4 5 6; do
  $RedisServerPath/redis-server $RedisConfPath/redis-700$i.conf
done

/usr/local/redis-cluster/bin/redis-trib.rb create --replicas 1 \
192.168.33.110:7001 \
192.168.33.110:7002 \
192.168.33.110:7003 \
192.168.33.110:7004 \
192.168.33.110:7005 \
192.168.33.110:7006 \
$ sudo chmod a+x /usr/local/redis-cluster/bin/start.sh
  • 停止腳本
$ sudo vim /usr/local/redis-cluster/bin/stop.sh
#!/usr/bin/env bash

RedisClusterPath=/usr/local/redis-cluster
RedisCliPath=/usr/local/redis-cluster/bin
ip=192.168.33.110

for i in 1 2 3 4 5 6; do
    $RedisCliPath/redis-cli -c -h $ip -p 700$i shutdown
done

rm -rf $RedisClusterPath/log/*
rm -rf $RedisClusterPath/run/*
rm -rf $RedisClusterPath/var/*
$ sudo chmod a+x /usr/local/redis-cluster/bin/stop.sh

查看信息

$ /usr/local/redis-cluster/bin/redis-cli -c -h 192.168.33.110 -p 7001
192.168.33.110:7001> cluster nodes # 查看節點信息
5521b06a46910c873e7ee8a274f2cb9fc41ffed1 192.168.33.110:7003@17003 master - 0 1526681994008 3 connected 10923-16383
8153a1b35e0fab1e48f0171653f6cffd82cbc5a6 192.168.33.110:7004@17004 slave d7e0395152496f41fe73fbbdabc8499c79f6632e 0 1526681996033 4 connected
a2ff29661185f6ad17c99232d0490b3814ba0164 192.168.33.110:7006@17006 slave affc237bf177fd4ac591314736a1358e16545d0c 0 1526681995020 6 connected
d7e0395152496f41fe73fbbdabc8499c79f6632e 192.168.33.110:7002@17002 master - 0 1526681993000 2 connected 5461-10922
8b613e4b5abbdf3c2486dfcb8bc1e9dde1897820 192.168.33.110:7005@17005 slave 5521b06a46910c873e7ee8a274f2cb9fc41ffed1 0 1526681993000 5 connected
affc237bf177fd4ac591314736a1358e16545d0c 192.168.33.110:7001@17001 myself,master - 0 1526681994000 1 connected 0-5460
192.168.33.110:7001> cluster info # 查看集羣信息
cluster_state:ok
cluster_slots_assigned:16384
cluster_slots_ok:16384
cluster_slots_pfail:0
cluster_slots_fail:0
cluster_known_nodes:6
cluster_size:3
cluster_current_epoch:6
cluster_my_epoch:1
cluster_stats_messages_ping_sent:59501
cluster_stats_messages_pong_sent:61755
cluster_stats_messages_sent:121256
cluster_stats_messages_ping_received:61750
cluster_stats_messages_pong_received:59501
cluster_stats_messages_meet_received:5
cluster_stats_messages_received:121256

相關命令

命令 説明
CLUSTER INFO 打印集羣信息
CLUSTER NODES 列出集羣當前已知的所有節點信息
CLUSTER MEET <ip> <port> 將 ip 和 port 所指定的節點添加到集羣當中
CLUSTER FORGET <node_id> 從集羣中移除 node_id 指定的節點
CLUSTER REPLICATE <node_id> 將當前節點設置為 node_id 指定的節點的從節點
CLUSTER SAVECONFIG 將節點的配置文件保存到硬盤裏面
CLUSTER ADDSLOTS <slot> [slot ...] 將一個或多個槽（slot）指派（assign）給當前節點
CLUSTER DELSLOTS <slot> [slot ...] 移除一個或多個槽對當前節點的指派
CLUSTER FLUSHSLOTS 移除指派給當前節點的所有槽，讓當前節點變成一個沒有指派任何槽的節點
CLUSTER SETSLOT <slot> NODE <node_id> 將槽 slot 指派給 node_id 指定的節點，如果槽已經指派給另一個節點，那麼先讓另一個節點刪除該槽，然後再進行指派
CLUSTER SETSLOT <slot> MIGRATING <node_id> 將本節點的槽 slot 遷移到 node_id 指定的節點中
CLUSTER SETSLOT <slot> IMPORTING <node_id> 從 node_id 指定的節點中導入槽 slot 到本節點
CLUSTER SETSLOT <slot> STABLE 取消對槽 slot 的導入（import）或者遷移（migrate）
CLUSTER KEYSLOT <key> 計算鍵 key 應該被放置在哪個槽上
CLUSTER COUNTKEYSINSLOT <slot> 返回槽 slot 目前包含的鍵值對數量
CLUSTER GETKEYSINSLOT <slot> <count> 返回 count 個 slot 槽中的鍵

實例操作

客户端操作

// 從某個節點設置值後，去不同的節點獲取設置的值
$ /usr/local/redis-cluster/bin/redis-cli -c -h 192.168.33.110 -p 7001
192.168.33.110:7001> set foo bar
-> Redirected to slot [12182] located at 192.168.33.110:7003
OK
$ /usr/local/redis-cluster/bin/redis-cli -c -h 192.168.33.110 -p 7001
192.168.33.110:7001> get foo
-> Redirected to slot [12182] located at 192.168.33.110:7003
"bar"
$ /usr/local/redis-cluster/bin/redis-cli -c -h 192.168.33.110 -p 7002
192.168.33.110:7002> get foo
-> Redirected to slot [12182] located at 192.168.33.110:7003
"bar"
$ /usr/local/redis-cluster/bin/redis-cli -c -h 192.168.33.110 -p 7003
192.168.33.110:7003> get foo
"bar"
$ /usr/local/redis-cluster/bin/redis-cli -c -h 192.168.33.110 -p 7004
192.168.33.110:7004> get foo
-> Redirected to slot [12182] located at 192.168.33.110:7003
"bar"
$ /usr/local/redis-cluster/bin/redis-cli -c -h 192.168.33.110 -p 7005
192.168.33.110:7005> get foo
-> Redirected to slot [12182] located at 192.168.33.110:7003
"bar"
$ /usr/local/redis-cluster/bin/redis-cli -c -h 192.168.33.110 -p 7006
192.168.33.110:7006> get foo
-> Redirected to slot [12182] located at 192.168.33.110:7003
"bar"

程序操作

下載安裝

  • Flexible and feature-complete Redis client for PHP and HHVM
  • Quick tour
$ cd ~
$ mkdir test
$ cd test
$ composer init
$ composer install
$ composer require predis/predis

測試代碼

<?php

require __DIR__ . '/vendor/autoload.php';

for ($i = 0; $i < 6; $i++) {
    $parameters[] = [
        'host'               => '192.168.33.110',
        'port'               => '700' . $i,
        'password'           => null,
        'database'           => 0,
        'timeout'            => 5,
        'read_write_timeout' => 5,
    ];
}
$options = ['cluster' => 'redis'];

try {
    $cluster = new \Predis\Client($parameters, $options);
} catch (\Predis\PredisException $e) {
    echo $e->getMessage();
    exit();
}

for ($i = 1; $i < 100; $i++) {
    $format = 'No.%s is a RedisCluster test.';
    $cluster->set("$i", sprintf($format, $i));
    echo $cluster->get("$i") . PHP_EOL;
}

測試結果

$ php cluster.php
No.1 is a Redis cluster test.
No.2 is a Redis cluster test.
No.3 is a Redis cluster test.
...
No.99 is a Redis cluster test.

$ /usr/local/redis-cluster/bin/redis-cli -c -h 192.168.33.110 -p 7001
192.168.33.110:7001> get 11
"No.11 is a RedisCluster test."
192.168.33.110:7001> get 23
-> Redirected to slot [9671] located at 192.168.33.110:7002
"No.23 is a RedisCluster test."
192.168.33.110:7002> get 33
"No.33 is a RedisCluster test."
192.168.33.110:7002> get 67
"No.67 is a RedisCluster test."
192.168.33.110:7002> get 88
-> Redirected to slot [15207] located at 192.168.33.110:7003
"No.88 is a RedisCluster test."
192.168.33.110:7003> get 99
-> Redirected to slot [6263] located at 192.168.33.110:7002
"No.99 is a RedisCluster test."
192.168.33.110:7002> get 100
-> Redirected to slot [339] located at 192.168.33.110:7001
(nil)
192.168.33.110:7001> get 17
-> Redirected to slot [12304] located at 192.168.33.110:7003
"No.17 is a RedisCluster test."
192.168.33.110:7003> get 42
-> Redirected to slot [8000] located at 192.168.33.110:7002
"No.42 is a RedisCluster test."
192.168.33.110:7002> get 56
-> Redirected to slot [11509] located at 192.168.33.110:7003
"No.56 is a RedisCluster test."
192.168.33.110:7003> get 61
-> Redirected to slot [2369] located at 192.168.33.110:7001
"No.61 is a RedisCluster test."
redis , redis集羣
user avatar
0 位用戶收藏了這個故事！

發佈 評論

Some HTML is okay.