starrocks三节点集群因服务器时间不同步导致配置失败

skylines 2024-03-23

477

通过MySQL客户端登录fe使用SHOW PROC '/frontends'\G查看starrocks高可用状态时候，发现集群的两个follow节点一直有异常。

看到两个follow都已经加入到集群里，但是Alive: false一直都是这样的状态，原因是follow节点的fe一直启动不成功。如下高可用信息以及状态显示

mysql> SHOW PROC '/frontends'\G
*************************** 1. row ***************************
             Name: 192.168.163.107_9010_1711093743940
               IP: 192.168.163.107
      EditLogPort: 9010
         HttpPort: 8030
        QueryPort: 0
          RpcPort: 0
             Role: FOLLOWER
        ClusterId: 1399291431
             Join: true
            Alive: false
ReplayedJournalId: 0
    LastHeartbeat: NULL
         IsHelper: true
           ErrMsg: got exception
        StartTime: NULL
          Version: NULL
*************************** 2. row ***************************
             Name: 192.168.163.106_9010_1711093742495
               IP: 192.168.163.106
      EditLogPort: 9010
         HttpPort: 8030
        QueryPort: 0
          RpcPort: 0
             Role: FOLLOWER
        ClusterId: 1399291431
             Join: true
            Alive: false
ReplayedJournalId: 0
    LastHeartbeat: NULL
         IsHelper: true
           ErrMsg: got exception
        StartTime: NULL
          Version: NULL
*************************** 3. row ***************************
             Name: 192.168.163.105_9010_1711076058445
               IP: 192.168.163.105
      EditLogPort: 9010
         HttpPort: 8030
        QueryPort: 9030
          RpcPort: 9020
             Role: LEADER
        ClusterId: 1399291431
             Join: true
            Alive: true
ReplayedJournalId: 5434
    LastHeartbeat: 2024-03-22 15:54:40
         IsHelper: true
           ErrMsg:
        StartTime: 2024-03-22 15:38:46
          Version: 3.2.1-79ee91d
3 rows in set (0.03 sec)
follow节点的ErrMsg: got exception。

查看了fe的配置参数以及java环境，尝试多次启动fe都不成功。然后，查看follow节点的fe日志，发现三台服务器之间的系统时间有问题。

2024-03-22 15:50:32,138 INFO (main|1) [StarRocksFE.start():129] StarRocks FE starting, version: 3.2.1-79ee91d
2024-03-22 15:50:32,145 INFO (main|1) [FrontendOptions.analyzePriorityCidrs():299] configured prior_cidrs value: 192.168.163.106
2024-03-22 15:50:32,147 INFO (main|1) [FrontendOptions.initAddrUseIp():249] Use IP init local addr, IP: 192.168.163.106
2024-03-22 15:50:32,466 INFO (main|1) [Auth.grantRoleInternal():837] grant operator to 'root'@'%', isReplay = true
2024-03-22 15:50:32,495 INFO (main|1) [AuthorizationMgr.initBuiltinRoleUnlocked():282] create built-in role root[-1]
2024-03-22 15:50:32,501 INFO (main|1) [AuthorizationMgr.initBuiltinRoleUnlocked():282] create built-in role db_admin[-2]
2024-03-22 15:50:32,502 INFO (main|1) [AuthorizationMgr.initBuiltinRoleUnlocked():282] create built-in role cluster_admin[-3]
2024-03-22 15:50:32,502 INFO (main|1) [AuthorizationMgr.initBuiltinRoleUnlocked():282] create built-in role user_admin[-4]
2024-03-22 15:50:32,502 INFO (main|1) [AuthorizationMgr.initBuiltinRoleUnlocked():282] create built-in role public[-5]
2024-03-22 15:50:32,503 INFO (main|1) [GlobalStateMgr.initAuth():1157] using new privilege framework..
2024-03-22 15:50:32,697 INFO (main|1) [NodeMgr.getHelperNodes():658] get helper nodes: [192.168.163.105:9010]
2024-03-22 15:50:32,762 INFO (main|1) [NodeMgr.getFeNodeTypeAndNameFromHelpers():513] get fe node type FOLLOWER, name 192.168.163.106_9010_1711093742495 from 192.168.163.105:8030
2024-03-22 15:50:32,772 INFO (main|1) [NodeMgr.getVersionFileFromHelper():711] Downloading version file from http://192.168.163.105:8030/version
2024-03-22 15:50:32,998 INFO (main|1) [NodeMgr.getNewImageOnStartup():741] start to download image.5143 from http://192.168.163.105:8030/image?version=5143&subdir=
2024-03-22 15:50:33,001 INFO (main|1) [NodeMgr.getClusterIdAndRoleOnStartup():459] Current run_mode is shared_nothing
2024-03-22 15:50:33,002 INFO (main|1) [NodeMgr.getClusterIdAndRoleOnStartup():466] Got cluster id: 1399291431, role: FOLLOWER, node name: 192.168.163.106_9010_1711093742495 and run_mode: shared_nothing
2024-03-22 15:50:33,003 INFO (main|1) [BDBEnvironment.ensureHelperInLocal():350] start to check if local replica environment from datanode2/starrocks/fe/meta/bdb contains 192.168.163.105:9010
2024-03-22 15:50:33,367 INFO (UNKNOWN 192.168.163.106_9010_1711093742495(-1)|1) [BDBEnvironment.ensureHelperInLocal():368] skip check empty environment
2024-03-22 15:50:33,398 INFO (UNKNOWN 192.168.163.106_9010_1711093742495(-1)|1) [BDBEnvironment.setupEnvironment():269] start to setup bdb environment for 1 times
2024-03-22 15:50:33,458 WARN (UNKNOWN 192.168.163.106_9010_1711093742495(-1)|1) [StateChangeExecutor.notifyNewFETypeTransfer():62] notify new FE type transfer: UNKNOWN
2024-03-22 15:50:33,508 INFO (UNKNOWN 192.168.163.106_9010_1711093742495(-1)|1) [BDBEnvironment.setupEnvironment():280] replicated environment is all set, wait for state change...


2024-03-22 15:50:33,508 WARN (RepNode 192.168.163.106_9010_1711093742495(-1)|83) [BDBStateChangeListener.stateChange():79] this node is DETACHED


2024-03-22 15:50:43,510 INFO (UNKNOWN 192.168.163.106_9010_1711093742495(-1)|1) [BDBEnvironment.setupEnvironment():288] state change done, current role UNKNOWN
2024-03-22 15:50:43,511 ERROR (UNKNOWN 192.168.163.106_9010_1711093742495(-1)|1) [BDBEnvironment.setupEnvironment():305] failed to setup environment after retried 1 times
com.sleepycat.je.EnvironmentFailureException: (JE 18.3.16) Environment must be closed, caused by: com.sleepycat.je.EnvironmentFailureException: Environment invalid because of previous exception: (JE 18.3.16) 192.168.163.106_9010_1711093742495(-1):/datanode2/starrocks/fe/meta/bdb Clock delta: -21269 ms. between Feeder: 192.168.163.105_9010_1711076058445 and this Replica exceeds max permissible delta: 5000 ms. HANDSHAKE_ERROR: Error during the handshake between two nodes. Some validity or compatibility check failed, preventing further communication between the nodes. Environment is invalid and must be closed. Originally thrown by HA thread: RepNode 192.168.163.106_9010_1711093742495(-1) Originally thrown by HA thread: RepNode 192.168.163.106_9010_1711093742495(-1)
        at com.sleepycat.je.EnvironmentFailureException.wrapSelf(EnvironmentFailureException.java:230) ~[starrocks-bdb-je-18.3.16.jar:?]
        at com.sleepycat.je.dbi.EnvironmentImpl.checkIfInvalid(EnvironmentImpl.java:1835) ~[starrocks-bdb-je-18.3.16.jar:?]
        at com.sleepycat.je.dbi.EnvironmentImpl.checkOpen(EnvironmentImpl.java:1844) ~[starrocks-bdb-je-18.3.16.jar:?]
        at com.sleepycat.je.Environment.checkOpen(Environment.java:2697) ~[starrocks-bdb-je-18.3.16.jar:?]
        at com.sleepycat.je.Environment.openDatabase(Environment.java:659) ~[starrocks-bdb-je-18.3.16.jar:?]
        at com.starrocks.journal.bdbje.BDBEnvironment.setupEnvironment(BDBEnvironment.java:291) ~[starrocks-fe.jar:?]
        at com.starrocks.journal.bdbje.BDBEnvironment.setup(BDBEnvironment.java:175) ~[starrocks-fe.jar:?]
        at com.starrocks.journal.bdbje.BDBEnvironment.initBDBEnvironment(BDBEnvironment.java:153) ~[starrocks-fe.jar:?]
        at com.starrocks.journal.JournalFactory.create(JournalFactory.java:31) ~[starrocks-fe.jar:?]
        at com.starrocks.server.GlobalStateMgr.initJournal(GlobalStateMgr.java:1181) ~[starrocks-fe.jar:?]
        at com.starrocks.server.GlobalStateMgr.initialize(GlobalStateMgr.java:1130) ~[starrocks-fe.jar:?]
        at com.starrocks.StarRocksFE.start(StarRocksFE.java:135) ~[starrocks-fe.jar:?]
        at com.starrocks.StarRocksFE.main(StarRocksFE.java:84) ~[starrocks-fe.jar:?]
Caused by: com.sleepycat.je.EnvironmentFailureException: Environment invalid because of previous exception: (JE 18.3.16) 192.168.163.106_9010_1711093742495(-1):/datanode2/starrocks/fe/meta/bdb Clock delta: -21269 ms. between Feeder
: 192.168.163.105_9010_1711076058445 and this Replica exceeds max permissible delta: 5000 ms. HANDSHAKE_ERROR: Error during the handshake between two nodes. Some validity or compatibility check failed, preventing further communication between the nodes. Environment is invalid and must be closed. Originally thrown by HA thread: RepNode 192.168.163.106_9010_1711093742495(-1) Originally thrown by HA thread: RepNode 192.168.163.106_9010_1711093742495(-1)
        at com.sleepycat.je.rep.stream.ReplicaFeederHandshake.checkClockSkew(ReplicaFeederHandshake.java:432) ~[starrocks-bdb-je-18.3.16.jar:?]
        at com.sleepycat.je.rep.stream.ReplicaFeederHandshake.execute(ReplicaFeederHandshake.java:269) ~[starrocks-bdb-je-18.3.16.jar:?]
        at com.sleepycat.je.rep.impl.node.Replica.initReplicaLoop(Replica.java:709) ~[starrocks-bdb-je-18.3.16.jar:?]
        at com.sleepycat.je.rep.impl.node.Replica.runReplicaLoopInternal(Replica.java:485) ~[starrocks-bdb-je-18.3.16.jar:?]
        at com.sleepycat.je.rep.impl.node.Replica.runReplicaLoop(Replica.java:412) ~[starrocks-bdb-je-18.3.16.jar:?]
        at com.sleepycat.je.rep.impl.node.RepNode.run(RepNode.java:1869) ~[starrocks-bdb-je-18.3.16.jar:?]
2024-03-22 15:50:43,519 INFO (UNKNOWN 192.168.163.106_9010_1711093742495(-1)|1) [BDBEnvironment.close():517] start to close epoch database
2024-03-22 15:50:43,519 INFO (UNKNOWN 192.168.163.106_9010_1711093742495(-1)|1) [BDBEnvironment.close():526] close epoch database end
2024-03-22 15:50:43,519 INFO (UNKNOWN 192.168.163.106_9010_1711093742495(-1)|1) [BDBEnvironment.close():528] start to close replicated environment
2024-03-22 15:50:43,521 INFO (UNKNOWN 192.168.163.106_9010_1711093742495(-1)|1) [BDBEnvironment.close():538] close replicated environment end
2024-03-22 15:50:48,522 INFO (UNKNOWN 192.168.163.106_9010_1711093742495(-1)|1) [BDBEnvironment.setupEnvironment():269] start to setup bdb environment for 2 times
2024-03-22 15:50:48,548 WARN (UNKNOWN 192.168.163.106_9010_1711093742495(-1)|1) [StateChangeExecutor.notifyNewFETypeTransfer():62] notify new FE type transfer: UNKNOWN
2024-03-22 15:50:48,549 INFO (UNKNOWN 192.168.163.106_9010_1711093742495(-1)|1) [BDBEnvironment.setupEnvironment():280] replicated environment is all set, wait for state change...
2024-03-22 15:50:48,557 WARN (RepNode 192.168.163.106_9010_1711093742495(-1)|104) [BDBStateChangeListener.stateChange():79] this node is DETACHED
2024-03-22 15:50:58,550 INFO (UNKNOWN 192.168.163.106_9010_1711093742495(-1)|1) [BDBEnvironment.setupEnvironment():288] state change done, current role UNKNOWN
2024-03-22 15:50:58,550 ERROR (UNKNOWN 192.168.163.106_9010_1711093742495(-1)|1) [BDBEnvironment.setupEnvironment():305] failed to setup environment after retried 2 times
com.sleepycat.je.EnvironmentFailureException: (JE 18.3.16) Environment must be closed, caused by: com.sleepycat.je.EnvironmentFailureException: Environment invalid because of previous exception: (JE 18.3.16) 192.168.163.106_9010_1711093742495(-1):/datanode2/starrocks/fe/meta/bdb Clock delta: -21270 ms. between Feeder: 192.168.163.105_9010_1711076058445 and this Replica exceeds max permissible delta: 5000 ms. HANDSHAKE_ERROR: Error during the handshake between two nodes. Some validity or compatibility check failed, preventing further communication between the nodes. Environment is invalid and must be closed. Originally thrown by HA thread: RepNode 192.168.163.106_9010_1711093742495(-1) Originally thrown by HA thread: RepNode 192.168.163.106_9010_1711093742495(-1)
        at com.sleepycat.je.EnvironmentFailureException.wrapSelf(EnvironmentFailureException.java:230) ~[starrocks-bdb-je-18.3.16.jar:?]
        at com.sleepycat.je.dbi.EnvironmentImpl.checkIfInvalid(EnvironmentImpl.java:1835) ~[starrocks-bdb-je-18.3.16.jar:?]
        at com.sleepycat.je.dbi.EnvironmentImpl.checkOpen(EnvironmentImpl.java:1844) ~[starrocks-bdb-je-18.3.16.jar:?]
        at com.sleepycat.je.Environment.checkOpen(Environment.java:2697) ~[starrocks-bdb-je-18.3.16.jar:?]
        at com.sleepycat.je.Environment.openDatabase(Environment.java:659) ~[starrocks-bdb-je-18.3.16.jar:?]
        at com.starrocks.journal.bdbje.BDBEnvironment.setupEnvironment(BDBEnvironment.java:291) ~[starrocks-fe.jar:?]
        at com.starrocks.journal.bdbje.BDBEnvironment.setup(BDBEnvironment.java:175) ~[starrocks-fe.jar:?]
        at com.starrocks.journal.bdbje.BDBEnvironment.initBDBEnvironment(BDBEnvironment.java:153) ~[starrocks-fe.jar:?]
        at com.starrocks.journal.JournalFactory.create(JournalFactory.java:31) ~[starrocks-fe.jar:?]
        at com.starrocks.server.GlobalStateMgr.initJournal(GlobalStateMgr.java:1181) ~[starrocks-fe.jar:?]
        at com.starrocks.server.GlobalStateMgr.initialize(GlobalStateMgr.java:1130) ~[starrocks-fe.jar:?]
        at com.starrocks.StarRocksFE.start(StarRocksFE.java:135) ~[starrocks-fe.jar:?]
        at com.starrocks.StarRocksFE.main(StarRocksFE.java:84) ~[starrocks-fe.jar:?]
Caused by: com.sleepycat.je.EnvironmentFailureException: Environment invalid because of previous exception: (JE 18.3.16) 192.168.163.106_9010_1711093742495(-1):/datanode2/starrocks/fe/meta/bdb Clock delta: -21270 ms. between Feeder: 192.168.163.105_9010_1711076058445 and this Replica exceeds max permissible delta: 5000 ms. HANDSHAKE_ERROR: Error during the handshake between two nodes. Some validity or compatibility check failed, preventing further communication between the nodes. Environment is invalid and must be closed. Originally thrown by HA thread: RepNode 192.168.163.106_9010_1711093742495(-1) Originally thrown by HA thread: RepNode 192.168.163.106_9010_1711093742495(-1)
        at com.sleepycat.je.rep.stream.ReplicaFeederHandshake.checkClockSkew(ReplicaFeederHandshake.java:432) ~[starrocks-bdb-je-18.3.16.jar:?]
        at com.sleepycat.je.rep.stream.ReplicaFeederHandshake.execute(ReplicaFeederHandshake.java:269) ~[starrocks-bdb-je-18.3.16.jar:?]
        at com.sleepycat.je.rep.impl.node.Replica.initReplicaLoop(Replica.java:709) ~[starrocks-bdb-je-18.3.16.jar:?]
        at com.sleepycat.je.rep.impl.node.Replica.runReplicaLoopInternal(Replica.java:485) ~[starrocks-bdb-je-18.3.16.jar:?]
        at com.sleepycat.je.rep.impl.node.Replica.runReplicaLoop(Replica.java:412) ~[starrocks-bdb-je-18.3.16.jar:?]
        at com.sleepycat.je.rep.impl.node.RepNode.run(RepNode.java:1869) ~[starrocks-bdb-je-18.3.16.jar:?]
2024-03-22 15:50:58,551 INFO (UNKNOWN 192.168.163.106_9010_1711093742495(-1)|1) [BDBEnvironment.close():517] start to close epoch database
2024-03-22 15:50:58,551 INFO (UNKNOWN 192.168.163.106_9010_1711093742495(-1)|1) [BDBEnvironment.close():526] close epoch database end
2024-03-22 15:50:58,551 INFO (UNKNOWN 192.168.163.106_9010_1711093742495(-1)|1) [BDBEnvironment.close():528] start to close replicated environment
2024-03-22 15:50:58,552 INFO (UNKNOWN 192.168.163.106_9010_1711093742495(-1)|1) [BDBEnvironment.close():538] close replicated environment end
2024-03-22 15:51:03,553 INFO (UNKNOWN 192.168.163.106_9010_1711093742495(-1)|1) [BDBEnvironment.setupEnvironment():269] start to setup bdb environment for 3 times
2024-03-22 15:51:03,582 WARN (UNKNOWN 192.168.163.106_9010_1711093742495(-1)|1) [StateChangeExecutor.notifyNewFETypeTransfer():62] notify new FE type transfer: UNKNOWN
2024-03-22 15:51:03,582 INFO (UNKNOWN 192.168.163.106_9010_1711093742495(-1)|1) [BDBEnvironment.setupEnvironment():280] replicated environment is all set, wait for state change...
2024-03-22 15:51:03,589 WARN (RepNode 192.168.163.106_9010_1711093742495(-1)|124) [BDBStateChangeListener.stateChange():79] this node is DETACHED
2024-03-22 15:51:13,583 INFO (UNKNOWN 192.168.163.106_9010_1711093742495(-1)|1) [BDBEnvironment.setupEnvironment():288] state change done, current role UNKNOWN
2024-03-22 15:51:13,584 ERROR (UNKNOWN 192.168.163.106_9010_1711093742495(-1)|1) [BDBEnvironment.setupEnvironment():305] failed to setup environment after retried 3 times
com.sleepycat.je.EnvironmentFailureException: (JE 18.3.16) Environment must be closed, caused by: com.sleepycat.je.EnvironmentFailureException: Environment invalid because of previous exception: (JE 18.3.16) 192.168.163.106_9010_1711093742495(-1):/datanode2/starrocks/fe/meta/bdb Clock delta: -21270 ms. between Feeder: 192.168.163.105_9010_1711076058445 and this Replica exceeds max permissible delta: 5000 ms. HANDSHAKE_ERROR: Error during the handshake between two nodes. Some validity or compatibility check failed, preventing further communication between the nodes. Environment is invalid and must be closed. Originally thrown by HA thread: RepNode 192.168.163.106_9010_1711093742495(-1) Originally thrown by HA thread: RepNode 192.168.163.106_9010_1711093742495(-1)
        at com.sleepycat.je.EnvironmentFailureException.wrapSelf(EnvironmentFailureException.java:230) ~[starrocks-bdb-je-18.3.16.jar:?]
        at com.sleepycat.je.dbi.EnvironmentImpl.checkIfInvalid(EnvironmentImpl.java:1835) ~[starrocks-bdb-je-18.3.16.jar:?]
        at com.sleepycat.je.dbi.EnvironmentImpl.checkOpen(EnvironmentImpl.java:1844) ~[starrocks-bdb-je-18.3.16.jar:?]
        at com.sleepycat.je.Environment.checkOpen(Environment.java:2697) ~[starrocks-bdb-je-18.3.16.jar:?]
        at com.sleepycat.je.Environment.openDatabase(Environment.java:659) ~[starrocks-bdb-je-18.3.16.jar:?]
        at com.starrocks.journal.bdbje.BDBEnvironment.setupEnvironment(BDBEnvironment.java:291) ~[starrocks-fe.jar:?]
        at com.starrocks.journal.bdbje.BDBEnvironment.setup(BDBEnvironment.java:175) ~[starrocks-fe.jar:?]
        at com.starrocks.journal.bdbje.BDBEnvironment.initBDBEnvironment(BDBEnvironment.java:153) ~[starrocks-fe.jar:?]
        at com.starrocks.journal.JournalFactory.create(JournalFactory.java:31) ~[starrocks-fe.jar:?]
        at com.starrocks.server.GlobalStateMgr.initJournal(GlobalStateMgr.java:1181) ~[starrocks-fe.jar:?]
        at com.starrocks.server.GlobalStateMgr.initialize(GlobalStateMgr.java:1130) ~[starrocks-fe.jar:?]
        at com.starrocks.StarRocksFE.start(StarRocksFE.java:135) ~[starrocks-fe.jar:?]
        at com.starrocks.StarRocksFE.main(StarRocksFE.java:84) ~[starrocks-fe.jar:?]
Caused by: com.sleepycat.je.EnvironmentFailureException: Environment invalid because of previous exception: (JE 18.3.16) 192.168.163.106_9010_1711093742495(-1):/datanode2/starrocks/fe/meta/bdb Clock delta: -21270 ms. between Feeder: 192.168.163.105_9010_1711076058445 and this Replica exceeds max permissible delta: 5000 ms. HANDSHAKE_ERROR: Error during the handshake between two nodes. Some validity or compatibility check failed, preventing further communication between the nodes. Environment is invalid and must be closed. Originally thrown by HA thread: RepNode 192.168.163.106_9010_1711093742495(-1) Originally thrown by HA thread: RepNode 192.168.163.106_9010_1711093742495(-1)
        at com.sleepycat.je.rep.stream.ReplicaFeederHandshake.checkClockSkew(ReplicaFeederHandshake.java:432) ~[starrocks-bdb-je-18.3.16.jar:?]
        at com.sleepycat.je.rep.stream.ReplicaFeederHandshake.execute(ReplicaFeederHandshake.java:269) ~[starrocks-bdb-je-18.3.16.jar:?]
        at com.sleepycat.je.rep.impl.node.Replica.initReplicaLoop(Replica.java:709) ~[starrocks-bdb-je-18.3.16.jar:?]
        at com.sleepycat.je.rep.impl.node.Replica.runReplicaLoopInternal(Replica.java:485) ~[starrocks-bdb-je-18.3.16.jar:?]
        at com.sleepycat.je.rep.impl.node.Replica.runReplicaLoop(Replica.java:412) ~[starrocks-bdb-je-18.3.16.jar:?]
        at com.sleepycat.je.rep.impl.node.RepNode.run(RepNode.java:1869) ~[starrocks-bdb-je-18.3.16.jar:?]
2024-03-22 15:51:13,584 INFO (UNKNOWN 192.168.163.106_9010_1711093742495(-1)|1) [BDBEnvironment.close():517] start to close epoch database
2024-03-22 15:51:13,584 INFO (UNKNOWN 192.168.163.106_9010_1711093742495(-1)|1) [BDBEnvironment.close():526] close epoch database end
2024-03-22 15:51:13,585 INFO (UNKNOWN 192.168.163.106_9010_1711093742495(-1)|1) [BDBEnvironment.close():528] start to close replicated environment
2024-03-22 15:51:13,586 INFO (UNKNOWN 192.168.163.106_9010_1711093742495(-1)|1) [BDBEnvironment.close():538] close replicated environment end
2024-03-22 15:51:13,586 ERROR (UNKNOWN 192.168.163.106_9010_1711093742495(-1)|1) [StarRocksFE.start():192] StarRocksFE start failed
com.starrocks.journal.JournalException: failed to setup environment after retried 3 times
        at com.starrocks.journal.bdbje.BDBEnvironment.setupEnvironment(BDBEnvironment.java:306) ~[starrocks-fe.jar:?]
        at com.starrocks.journal.bdbje.BDBEnvironment.setup(BDBEnvironment.java:175) ~[starrocks-fe.jar:?]
        at com.starrocks.journal.bdbje.BDBEnvironment.initBDBEnvironment(BDBEnvironment.java:153) ~[starrocks-fe.jar:?]
        at com.starrocks.journal.JournalFactory.create(JournalFactory.java:31) ~[starrocks-fe.jar:?]
        at com.starrocks.server.GlobalStateMgr.initJournal(GlobalStateMgr.java:1181) ~[starrocks-fe.jar:?]
        at com.starrocks.server.GlobalStateMgr.initialize(GlobalStateMgr.java:1130) ~[starrocks-fe.jar:?]
        at com.starrocks.StarRocksFE.start(StarRocksFE.java:135) ~[starrocks-fe.jar:?]
        at com.starrocks.StarRocksFE.main(StarRocksFE.java:84) ~[starrocks-fe.jar:?]
Caused by: com.sleepycat.je.EnvironmentFailureException: (JE 18.3.16) Environment must be closed, caused by: com.sleepycat.je.EnvironmentFailureException: Environment invalid because of previous exception: (JE 18.3.16) 192.168.163.106_9010_1711093742495(-1):/datanode2/starrocks/fe/meta/bdb Clock delta: -21270 ms. between Feeder: 192.168.163.105_9010_1711076058445 and this Replica exceeds max permissible delta: 5000 ms. HANDSHAKE_ERROR: Error during the handshake between two nodes. Some validity or compatibility check failed, preventing further communication between the nodes. Environment is invalid and must be closed. Originally thrown by HA thread: RepNode 192.168.163.106_9010_1711093742495(-1) Originally thrown by HA thread: RepNode 192.168.163.106_9010_1711093742495(-1)
        at com.sleepycat.je.EnvironmentFailureException.wrapSelf(EnvironmentFailureException.java:230) ~[starrocks-bdb-je-18.3.16.jar:?]
        at com.sleepycat.je.dbi.EnvironmentImpl.checkIfInvalid(EnvironmentImpl.java:1835) ~[starrocks-bdb-je-18.3.16.jar:?]
        at com.sleepycat.je.dbi.EnvironmentImpl.checkOpen(EnvironmentImpl.java:1844) ~[starrocks-bdb-je-18.3.16.jar:?]
        at com.sleepycat.je.Environment.checkOpen(Environment.java:2697) ~[starrocks-bdb-je-18.3.16.jar:?]
        at com.sleepycat.je.Environment.openDatabase(Environment.java:659) ~[starrocks-bdb-je-18.3.16.jar:?]
        at com.starrocks.journal.bdbje.BDBEnvironment.setupEnvironment(BDBEnvironment.java:291) ~[starrocks-fe.jar:?]
        ... 7 more
Caused by: com.sleepycat.je.EnvironmentFailureException: Environment invalid because of previous exception: (JE 18.3.16) 192.168.163.106_9010_1711093742495(-1):/datanode2/starrocks/fe/meta/bdb Clock delta: -21270 ms. between Feeder: 192.168.163.105_9010_1711076058445 and this Replica exceeds max permissible delta: 5000 ms. HANDSHAKE_ERROR: Error during the handshake between two nodes. Some validity or compatibility check failed, preventing further communication between the nodes. Environment is invalid and must be closed. Originally thrown by HA thread: RepNode 192.168.163.106_9010_1711093742495(-1) Originally thrown by HA thread: RepNode 192.168.163.106_9010_1711093742495(-1)
        at com.sleepycat.je.rep.stream.ReplicaFeederHandshake.checkClockSkew(ReplicaFeederHandshake.java:432) ~[starrocks-bdb-je-18.3.16.jar:?]
        at com.sleepycat.je.rep.stream.ReplicaFeederHandshake.execute(ReplicaFeederHandshake.java:269) ~[starrocks-bdb-je-18.3.16.jar:?]
        at com.sleepycat.je.rep.impl.node.Replica.initReplicaLoop(Replica.java:709) ~[starrocks-bdb-je-18.3.16.jar:?]
        at com.sleepycat.je.rep.impl.node.Replica.runReplicaLoopInternal(Replica.java:485) ~[starrocks-bdb-je-18.3.16.jar:?]
        at com.sleepycat.je.rep.impl.node.Replica.runReplicaLoop(Replica.java:412) ~[starrocks-bdb-je-18.3.16.jar:?]
        at com.sleepycat.je.rep.impl.node.RepNode.run(RepNode.java:1869) ~[starrocks-bdb-je-18.3.16.jar:?]

从上面日志可以看到多行这样的信息：

Caused by: com.sleepycat.je.EnvironmentFailureException: Environment invalid because of previous exception: (JE 18.3.16) 192.168.163.106_9010_1711093742495(-1):/datanode2/starrocks/fe/meta/bdb Clock delta: -21270 ms. between Feeder: 192.168.163.105_9010_1711076058445 and this Replica exceeds max permissible delta: 5000 ms. HANDSHAKE_ERROR: Error during the handshake between two nodes. Some validity or compatibility check failed, preventing further communication between the nodes. Environment is invalid and must be closed. Originally thrown by HA thread: RepNode 192.168.163.106_9010_1711093742495(-1) Originally thrown by HA thread: RepNode 192.168.163.106_9010_1711093742495(-1)

两个follow节点系统时间一样，但是比leader节点的系统时间（北京时间）快了20多秒，超过了高可用约束的同步时间5000ms.

解决方法：

后面就将两个follow节点的操作时间调节回北京时间，再启动它们的fe，最后fe启动成功，高可用配置成功，Alive的状态均转为 true。

info

文章转载自skylines，如果涉嫌侵权，请发送邮件至：contact@modb.pro进行举报，并提供相关证据，一经查实，墨天轮将立刻删除相关内容。

starrocks三节点集群因服务器时间不同步导致配置失败

评论