redis的数据迁移

数据库这点小事 2019-11-07

1680

redis -> redis

redis到redis的迁移我们可以使用最简单的迁移rdb文件的方式来实现

redis的备份rdb文件方式有两种，save和bgsave

save：阻塞主进程，客户端无法连接redis,等save完成之后，主进程才开始工作

bgsave: 通过fork一个save的子进程，在执行save过程中，不影响主进程，客户端可以正常连接redis

查看备份目录

[root@zookeeper-1 database]# redis-cli -p 45001
127.0.0.1:45001> config get dir
1) "dir"
2) "/appl/redis-2.8.21/database"

备份最新的redis数据

127.0.0.1:45001> bgsave
Background saving started

bgsave指令会立刻返回结果，save过程在后台完成，不过需要实时关注rdb文件，当文件大小不在变化是备份完成

迁移过程：将备份下的rdb文件传到需要恢复的redis服务器上，修改目标服务器上redis配置文件中rdb文件命名，和备份的rdb文件保持一致，然后启动目标服务器redis服务。完成redis->redis的迁移

redis->codis

codis是分布式架构，redis到codis的数据迁移我们利用redis-proxy完成

安装redis-port

下载redis-port源码包：https://github.com/CodisLabs/redis-port

源码编译一直有问题，没解决掉，退而求其次，下了编译好的

https://github.com/CodisLabs/redis-port/releases

解压，更改目录，配置环境变量。

tar -zxvf redis-port-v2.0-beta-go1.10.1-linux.tar.gz
mv redis-port-v2.0-beta-go1.10.1-linux usr/local/redis-port

redis-port下有4个命令：

redis-decode，redis-dump，redis-restore，redis-sync

decode 是把rdb文件解码输出到文件

restore 是把rdb文件传输到指定redis实例

dump 是从redis实例数据转存为rdb文件

sync 是两个redis实例间数据同步（rewrite存在相同的key覆盖）

[root@localhost local]# redis-decode -h
Usage:
        redis-decode [--ncpu=N] [--input=INPUT|INPUT] [--output=OUTPUT]
        redis-decode  --version
Options:
        -n N, --ncpu=N                    Set runtime.GOMAXPROCS to N.
        -i INPUT, --input=INPUT           Set input rdb encoded file.  [default: dev/stdin].
        -o OUTPUT, --output=OUTPUT        Set output file. [default: dev/stdout].
Examples:
        $ redis-decode -i dump.rdb -o dump.log
        $ redis-decode    dump.rdb -o dump.log
        $ cat dump.rdb | redis-decode --ncpu=8 > dump.log

[root@localhost local]# redis-sync -h
Usage:
        redis-sync [--ncpu=N] (--master=MASTER|MASTER) --target=TARGET [--db=DB] [--tmpfile-size=SIZE [--tmpfile=FILE]]
        redis-sync  --version
Options:
        -n N, --ncpu=N                    Set runtime.GOMAXPROCS to N.
        -m MASTER, --master=MASTER        The master redis instance ([auth@]host:port).
        -t TARGET, --target=TARGET        The target redis instance ([auth@]host:port).
        --db=DB                           Accept db = DB, default is *.
        --tmpfile=FILE                    Use FILE to as socket buffer.
        --tmpfile-size=SIZE               Set FILE size. If no --tmpfile is provided, a temporary file under current folder will be created.
Examples:
        $ redis-sync -m 127.0.0.1:6379 -t 127.0.0.1:6380
        $ redis-sync    127.0.0.1:6379 -t passwd@127.0.0.1:6380
        $ redis-sync    127.0.0.1:6379 -t passwd@127.0.0.1:6380 --db=0
        $ redis-sync    127.0.0.1:6379 -t passwd@127.0.0.1:6380 --db=0 --tmpfile-size=10gb
        $ redis-sync    127.0.0.1:6379 -t passwd@127.0.0.1:6380 --db=0 --tmpfile-size=10gb --tmpfile ~/sockfile.tmp

[root@localhost local]# redis-dump -h
Usage:
        redis-dump [--ncpu=N] (--master=MASTER|MASTER) [--output=OUTPUT] [--aof=FILE]
        redis-dump  --version
Options:
        -n N, --ncpu=N                    Set runtime.GOMAXPROCS to N.
        -m MASTER, --master=MASTER        The master redis instance ([auth@]host:port).
        -o OUTPUT, --output=OUTPUT        Set output file. [default: dev/stdout].
        -a FILE, --aof=FILE               Also dump the replication backlog.
Examples:
        $ redis-dump    127.0.0.1:6379 -o dump.rdb
        $ redis-dump    127.0.0.1:6379 -o dump.rdb -a
        $ redis-dump -m passwd@192.168.0.1:6380 -o dump.rdb -a dump.aof

[root@localhost local]# redis-restore -h
Usage:
        redis-restore [--ncpu=N] [--input=INPUT|INPUT] --target=TARGET [--aof=FILE] [--db=DB] [--unixtime-in-milliseconds=EXPR]
        redis-restore  --version
Options:
        -n N, --ncpu=N                    Set runtime.GOMAXPROCS to N.
        -i INPUT, --input=INPUT           Set input rdb encoded file.
        -t TARGET, --target=TARGET        The target redis instance ([auth@]host:port).
        -a FILE, --aof=FILE               Also restore the replication backlog.
        --db=DB                           Accept db = DB, default is *.
        --unixtime-in-milliseconds=EXPR   Update expire time when restoring objects from RDB.
Examples:
        $ redis-restore    dump.rdb -t 127.0.0.1:6379
        $ redis-restore -i dump.rdb -t 127.0.0.1:6379 --aof dump.aof --db=1
        $ redis-restore             -t 127.0.0.1:6379 --aof dump.aof
        $ redis-restore             -t 127.0.0.1:6379 --db=0
        $ redis-restore -i dump.rdb -t 127.0.0.1:6379 --unixtime-in-milliseconds="@209059200000"        ttlms += (now - '1976-08-17')
        $ redis-restore -i dump.rdb -t 127.0.0.1:6379 --unixtime-in-milliseconds="+1000"               / ttlms += 1s
        $ redis-restore -i dump.rdb -t 127.0.0.1:6379 --unixtime-in-milliseconds="-1000"               // ttlms -= 1s
        $ redis-restore -i dump.rdb -t 127.0.0.1:6379 --unixtime-in-milliseconds="1976-08-17 00:00:00" // ttlms += (now - '1976-08-17')

自动同步（sync）

报错1：

2019/11/06 19:42:40 libs.go:163: [PANIC] error response = "DENIED Redis is running in protected mode because protected mode is enabled, no bind address was specified, no authentication password is requested to clients. In this mode connections are only accepted from the loopback interface. If you want to connect from external computers to Redis you may adopt one of the following solutions: 1) Just disable protected mode sending the command 'CONFIG SET protected-mode no' from the loopback interface by connecting to Redis from the same host the server is running, however MAKE SURE Redis is not publicly accessible from internet if you do so. Use CONFIG REWRITE to make this change permanent. 2) Alternatively you can just disable the protected mode by editing the Redis configuration file, and setting the protected mode option to 'no', and then restarting the server. 3) If you started the server manually just for testing, restart it with the '--protected-mode no' option. 4) Setup a bind address or an authentication password. NOTE: You only need to do one of the above things in order for the server to start accepting connections from the outside."
[stack]:
    2   /home/travis/gopath/src/github.com/CodisLabs/redis-port/cmd/libs.go:163
            main.redisGetResponse
    1   /home/travis/gopath/src/github.com/CodisLabs/redis-port/cmd/libs.go:89
            main.redisSendPsyncFullsync
    0   /home/travis/gopath/src/github.com/CodisLabs/redis-port/cmd/sync.go:95
            main.main
        ... ...

解决：配置文件中protected-mode改为no

protected-mode no

重启redis

报错2：

2019/11/06 19:48:53 sync.go:76: [INFO] sync: master = "172.19.0.168:6379", target = "172.19.14.22:19000"
2019/11/06 19:48:53 sync.go:103: [INFO] +
2019/11/06 19:48:53 sync.go:109: [INFO] sync: runid = "8fe50c284493cdce1d359e1aadd43d06e332d5c5", offset = 0
2019/11/06 19:48:53 sync.go:110: [INFO] sync: rdb file = 24593444 (23.45mb)
2019/11/06 19:48:53 sync.go:208: [INFO] sync: (r/f,s/f,s) = (read,rdb.forward,rdb.skip/rdb.forward,rdb.skip)
2019/11/06 19:48:53 loader.go:50: [PANIC] Can't handle RDB format version = 9.
[stack]:
    1   /home/travis/gopath/src/github.com/CodisLabs/redis-port/pkg/rdb/loader.go:50
            github.com/CodisLabs/redis-port/pkg/rdb.(*Loader).Header
    0   /home/travis/gopath/src/github.com/CodisLabs/redis-port/cmd/libs.go:242
            main.newRDBLoader.func1
        ... ...

不同版本中的rdb格式不一样，redis-sync处理不了，很可惜，该命令的使用场景更多的是在同版本的redis下

PS:从redis迁移到codis时target地址选用codis其中一个proxy地址即可

不管是redis-sync还是redis-retore测试结果都不支持从高版本到低版本，低版本到高版本没问题

redis

文章转载自数据库这点小事，如果涉嫌侵权，请发送邮件至：contact@modb.pro进行举报，并提供相关证据，一经查实，墨天轮将立刻删除相关内容。

redis的数据迁移

评论