由于断电所有节点宕机,应用连接着主库的IP,然后做了异常操作创建了无主键的表并且做了dml操作,导致从节点无法加入集群。
故障现象:

错误日志如下:
[root@pg-1 ~]# tail -f /var/log/mysqld.log
2023-04-27T08:56:25.014944Z 36 [System] [MY-014002] [Repl] Replica receiver thread for channel 'group_replication_recovery': connected to source 'repl@192.168.207.131:3306' with server_uuid=bed33e2c-c883-11ed-b4ea-de4fc9577006, server_id=6. Starting GTID-based replication.
2023-04-27T08:56:25.463221Z 38 [ERROR] [MY-011542] [Repl] Plugin group_replication reported: 'Table jj does not have any PRIMARY KEY. This is not compatible with Group Replication.'
2023-04-27T08:56:25.463349Z 38 [ERROR] [MY-010584] [Repl] Replica SQL for channel 'group_replication_recovery': Worker 1 failed executing transaction 'bed33e2c-c883-11ed-b4ea-de4fc9577006:3' at source log binlog-bin.000026, end_log_pos 793; The table in transaction bed33e2c-c883-11ed-b4ea-de4fc9577006:3 does not comply with the requirements by an external plugin. Error_code: MY-003635
2023-04-27T08:56:25.464419Z 37 [ERROR] [MY-010586] [Repl] Error running query, replica SQL thread aborted. Fix the problem, and restart the replica SQL thread with "START REPLICA". We stopped at log 'binlog-bin.000026' position 573
2023-04-27T08:56:25.642472Z 35 [System] [MY-010597] [Repl] 'CHANGE REPLICATION SOURCE TO FOR CHANNEL 'group_replication_recovery' executed'. Previous state source_host='192.168.207.131', source_port= 3306, source_log_file='', source_log_pos= 4, source_bind=''. New state source_host='192.168.207.132', source_port= 3306, source_log_file='', source_log_pos= 4, source_bind=''.
2023-04-27T08:56:25.686537Z 42 [Warning] [MY-010897] [Repl] Storing MySQL user name or password information in the connection metadata repository is not secure and is therefore not recommended. Please consider using the USER and PASSWORD connection options for START REPLICA; see the 'START REPLICA Syntax' in the MySQL Manual for more information.
2023-04-27T08:56:25.710440Z 42 [System] [MY-014002] [Repl] Replica receiver thread for channel 'group_replication_recovery': connected to source 'repl@192.168.207.132:3306' with server_uuid=b893257d-c885-11ed-8be0-6a62c8446dfa, server_id=7. Starting GTID-based replication.
2023-04-27T08:56:25.715374Z 42 [ERROR] [MY-010557] [Repl] Error reading packet from server for channel 'group_replication_recovery': Cannot replicate because the source purged required binary logs. Replicate the missing transactions from elsewhere, or provision a new replica from backup. Consider increasing the source's binary log expiration period. The GTID set sent by the replica is 'aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaa:1-39648:1000012-1000023:2000019, bed33e2c-c883-11ed-b4ea-de4fc9577006:1-2', and the missing transactions are 'aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaa:39649-39652, bed33e2c-c883-11ed-b4ea-de4fc9577006:3' (server_errno=1236)
2023-04-27T08:56:25.715498Z 42 [ERROR] [MY-013114] [Repl] Replica I/O for channel 'group_replication_recovery': Got fatal error 1236 from source when reading data from binary log: 'Cannot replicate because the source purged required binary logs. Replicate the missing transactions from elsewhere, or provision a new replica from backup. Consider increasing the source's binary log expiration period. The GTID set sent by the replica is 'aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaa:1-39648:1000012-1000023:2000019, bed33e2c-c883-11ed-b4ea-de4fc9577006:1-2', and the missing transactions are 'aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaa:39649-39652, bed33e2c-c883-11ed-b4ea-de4fc9577006:3'', Error_code: MY-013114
2023-04-27T08:56:25.842367Z 21 [System] [MY-014010] [Repl] Plugin group_replication reported: 'Plugin 'group_replication' has been started.'
2023-04-27T08:57:25.802788Z 35 [System] [MY-010597] [Repl] 'CHANGE REPLICATION SOURCE TO FOR CHANNEL 'group_replication_recovery' executed'. Previous state source_host='192.168.207.132', source_port= 3306, source_log_file='', source_log_pos= 4, source_bind=''. New state source_host='192.168.207.132', source_port= 3306, source_log_file='', source_log_pos= 4, source_bind=''.
2023-04-27T08:57:25.842623Z 48 [Warning] [MY-010897] [Repl] Storing MySQL user name or password information in the connection metadata repository is not secure and is therefore not recommended. Please consider using the USER and PASSWORD connection options for START REPLICA; see the 'START REPLICA Syntax' in the MySQL Manual for more information.
2023-04-27T08:57:25.851681Z 48 [System] [MY-014002] [Repl] Replica receiver thread for channel 'group_replication_recovery': connected to source 'repl@192.168.207.132:3306' with server_uuid=b893257d-c885-11ed-8be0-6a62c8446dfa, server_id=7. Starting GTID-based replication.
2023-04-27T08:57:25.852429Z 48 [ERROR] [MY-010557] [Repl] Error reading packet from server for channel 'group_replication_recovery': Cannot replicate because the source purged required binary logs. Replicate the missing transactions from elsewhere, or provision a new replica from backup. Consider increasing the source's binary log expiration period. The GTID set sent by the replica is 'aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaa:1-39648:1000012-1000023:2000019, bed33e2c-c883-11ed-b4ea-de4fc9577006:1-2', and the missing transactions are 'aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaa:39649-39652, bed33e2c-c883-11ed-b4ea-de4fc9577006:3' (server_errno=1236)
2023-04-27T08:57:25.852623Z 48 [ERROR] [MY-013114] [Repl] Replica I/O for channel 'group_replication_recovery': Got fatal error 1236 from source when reading data from binary log: 'Cannot replicate because the source purged required binary logs. Replicate the missing transactions from elsewhere, or provision a new replica from backup. Consider increasing the source's binary log expiration period. The GTID set sent by the replica is 'aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaa:1-39648:1000012-1000023:2000019, bed33e2c-c883-11ed-b4ea-de4fc9577006:1-2', and the missing transactions are 'aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaa:39649-39652, bed33e2c-c883-11ed-b4ea-de4fc9577006:3'', Error_code: MY-013114
2023-04-27T08:57:25.952202Z 35 [System] [MY-010597] [Repl] 'CHANGE REPLICATION SOURCE TO FOR CHANNEL 'group_replication_recovery' executed'. Previous state source_host='192.168.207.132', source_port= 3306, source_log_file='', source_log_pos= 4, source_bind=''. New state source_host='192.168.207.131', source_port= 3306, source_log_file='', source_log_pos= 4, source_bind=''.
2023-04-27T08:57:26.086145Z 54 [Warning] [MY-010897] [Repl] Storing MySQL user name or password information in the connection metadata repository is not secure and is therefore not recommended. Please consider using the USER and PASSWORD connection options for START REPLICA; see the 'START REPLICA Syntax' in the MySQL Manual for more information.
2023-04-27T08:57:26.099064Z 54 [System] [MY-014002] [Repl] Replica receiver thread for channel 'group_replication_recovery': connected to source 'repl@192.168.207.131:3306' with server_uuid=bed33e2c-c883-11ed-b4ea-de4fc9577006, server_id=6. Starting GTID-based replication.
2023-04-27T08:57:26.141464Z 56 [ERROR] [MY-011542] [Repl] Plugin group_replication reported: 'Table jj does not have any PRIMARY KEY. This is not compatible with Group Replication.'
2023-04-27T08:57:26.141557Z 56 [ERROR] [MY-010584] [Repl] Replica SQL for channel 'group_replication_recovery': Worker 1 failed executing transaction 'bed33e2c-c883-11ed-b4ea-de4fc9577006:3' at source log binlog-bin.000026, end_log_pos 793; The table in transaction bed33e2c-c883-11ed-b4ea-de4fc9577006:3 does not comply with the requirements by an external plugin. Error_code: MY-003635
2023-04-27T08:57:26.142607Z 55 [ERROR] [MY-010586] [Repl] Error running query, replica SQL thread aborted. Fix the problem, and restart the replica SQL thread with "START REPLICA". We stopped at log 'binlog-bin.000026' position 573
2023-04-27T08:58:26.240949Z 35 [System] [MY-010597] [Repl] 'CHANGE REPLICATION SOURCE TO FOR CHANNEL 'group_replication_recovery' executed'. Previous state source_host='192.168.207.131', source_port= 3306, source_log_file='', source_log_pos= 4, source_bind=''. New state source_host='192.168.207.132', source_port= 3306, source_log_file='', source_log_pos= 4, source_bind=''.
2023-04-27T08:58:26.277761Z 60 [Warning] [MY-010897] [Repl] Storing MySQL user name or password information in the connection metadata repository is not secure and is therefore not recommended. Please consider using the USER and PASSWORD connection options for START REPLICA; see the 'START REPLICA Syntax' in the MySQL Manual for more information.
2023-04-27T08:58:26.296328Z 60 [System] [MY-014002] [Repl] Replica receiver thread for channel 'group_replication_recovery': connected to source 'repl@192.168.207.132:3306' with server_uuid=b893257d-c885-11ed-8be0-6a62c8446dfa, server_id=7. Starting GTID-based replication.
2023-04-27T08:58:26.297228Z 60 [ERROR] [MY-010557] [Repl] Error reading packet from server for channel 'group_replication_recovery': Cannot replicate because the source purged required binary logs. Replicate the missing transactions from elsewhere, or provision a new replica from backup. Consider increasing the source's binary log expiration period. The GTID set sent by the replica is 'aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaa:1-39648:1000012-1000023:2000019, bed33e2c-c883-11ed-b4ea-de4fc9577006:1-2', and the missing transactions are 'aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaa:39649-39652, bed33e2c-c883-11ed-b4ea-de4fc9577006:3' (server_errno=1236)
2023-04-27T08:58:26.297279Z 60 [ERROR] [MY-013114] [Repl] Replica I/O for channel 'group_replication_recovery': Got fatal error 1236 from source when reading data from binary log: 'Cannot replicate because the source purged required binary logs. Replicate the missing transactions from elsewhere, or provision a new replica from backup. Consider increasing the source's binary log expiration period. The GTID set sent by the replica is 'aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaa:1-39648:1000012-1000023:2000019, bed33e2c-c883-11ed-b4ea-de4fc9577006:1-2', and the missing transactions are 'aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaa:39649-39652, bed33e2c-c883-11ed-b4ea-de4fc9577006:3'', Error_code: MY-013114
2023-04-27T08:58:26.410388Z 35 [System] [MY-010597] [Repl] 'CHANGE REPLICATION SOURCE TO FOR CHANNEL 'group_replication_recovery' executed'. Previous state source_host='192.168.207.132', source_port= 3306, source_log_file='', source_log_pos= 4, source_bind=''. New state source_host='192.168.207.131', source_port= 3306, source_log_file='', source_log_pos= 4, source_bind=''.
2023-04-27T08:58:26.463584Z 66 [Warning] [MY-010897] [Repl] Storing MySQL user name or password information in the connection metadata repository is not secure and is therefore not recommended. Please consider using the USER and PASSWORD connection options for START REPLICA; see the 'START REPLICA Syntax' in the MySQL Manual for more information.
2023-04-27T08:58:26.499592Z 66 [System] [MY-014002] [Repl] Replica receiver thread for channel 'group_replication_recovery': connected to source 'repl@192.168.207.131:3306' with server_uuid=bed33e2c-c883-11ed-b4ea-de4fc9577006, server_id=6. Starting GTID-based replication.
2023-04-27T08:58:26.519397Z 68 [ERROR] [MY-011542] [Repl] Plugin group_replication reported: 'Table jj does not have any PRIMARY KEY. This is not compatible with Group Replication.'
2023-04-27T08:58:26.519500Z 68 [ERROR] [MY-010584] [Repl] Replica SQL for channel 'group_replication_recovery': Worker 1 failed executing transaction 'bed33e2c-c883-11ed-b4ea-de4fc9577006:3' at source log binlog-bin.000026, end_log_pos 793; The table in transaction bed33e2c-c883-11ed-b4ea-de4fc9577006:3 does not comply with the requirements by an external plugin. Error_code: MY-003635
2023-04-27T08:58:26.521029Z 67 [ERROR] [MY-010586] [Repl] Error running query, replica SQL thread aborted. Fix the problem, and restart the replica SQL thread with "START REPLICA". We stopped at log 'binlog-bin.000026' position 573
尝试指定gtid参数处理,但存在数据丢失,所以一般情况下不用,除非确认保证数据是一致的。
主库参考gtid
mysql> show global variables like '%gtid%';
+----------------------------------------------+----------------------------------------------------------------------------------------------------------------+
| Variable_name | Value |
+----------------------------------------------+----------------------------------------------------------------------------------------------------------------+
| binlog_gtid_simple_recovery | ON |
| enforce_gtid_consistency | ON |
| group_replication_gtid_assignment_block_size | 1000000 |
| gtid_executed | aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaa:1-39651:1000012-1000023:2000019,
bed33e2c-c883-11ed-b4ea-de4fc9577006:1-3 |
从库根据主库的gtid_executed值,执行:set global gtid_purged,注意在从库是gtid_purged不是gtid_executed,区别于主库。
mysql> show variables like '%gtid%';
+----------------------------------------------+----------------------------------------------------------------------------------------------------------------+
| Variable_name | Value |
+----------------------------------------------+----------------------------------------------------------------------------------------------------------------+
| binlog_gtid_simple_recovery | ON |
| enforce_gtid_consistency | ON |
| group_replication_gtid_assignment_block_size | 1000000 |
| gtid_executed | aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaa:1-39648:1000012-1000023:2000019,
bed33e2c-c883-11ed-b4ea-de4fc9577006:1-2 |
| gtid_executed_compression_period | 0 |
| gtid_mode | ON |
| gtid_next | AUTOMATIC |
| gtid_owned | |
| gtid_purged | aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaa:1-13 |
| session_track_gtids | OFF |
+----------------------------------------------+----------------------------------------------------------------------------------------------------------------+
10 rows in set (0.01 sec)
mysql> reset master;
Query OK, 0 rows affected (0.04 sec)
mysql> set global gtid_purged='aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaa:1-39651:1000012-1000023:2000019,bed33e2c-c883-11ed-b4ea-de4fc9577006:1-3';
Query OK, 0 rows affected (0.01 sec)
mysql> start group_replication;
Query OK, 0 rows affected (2.26 sec)
mysql> select * from performance_schema.replication_group_members;
+---------------------------+--------------------------------------+-----------------+-------------+--------------+-------------+----------------+----------------------------+
| CHANNEL_NAME | MEMBER_ID | MEMBER_HOST | MEMBER_PORT | MEMBER_STATE | MEMBER_ROLE | MEMBER_VERSION | MEMBER_COMMUNICATION_STACK |
+---------------------------+--------------------------------------+-----------------+-------------+--------------+-------------+----------------+----------------------------+
| group_replication_applier | b893257d-c885-11ed-8be0-6a62c8446dfa | 192.168.207.132 | 3306 | ONLINE | SECONDARY | 8.0.33 | XCom |
| group_replication_applier | bed33e2c-c883-11ed-b4ea-de4fc9577006 | 192.168.207.131 | 3306 | ONLINE | PRIMARY | 8.0.33 | XCom |
+---------------------------+--------------------------------------+-----------------+-------------+--------------+-------------+----------------+----------------------------+
2 rows in set (0.00 sec)
虽然online,但显示丢失事务的日志如下:
2023-04-27T08:36:05.658285Z 188 [ERROR] [MY-011542] [Repl] Plugin group_replication reported: 'Table jj does not have any PRIMARY KEY. This is not compatible with Group Replication.'
2023-04-27T08:36:05.658389Z 188 [ERROR] [MY-010584] [Repl] Replica SQL for channel 'group_replication_recovery': Worker 1 failed executing transaction 'bed33e2c-c883-11ed-b4ea-de4fc9577006:3' at source log binlog-bin.000026, end_log_pos 793; The table in transaction bed33e2c-c883-11ed-b4ea-de4fc9577006:3 does not comply with the requirements by an external plugin. Error_code: MY-003635
2023-04-27T08:36:05.659458Z 187 [ERROR] [MY-010586] [Repl] Error running query, replica SQL thread aborted. Fix the problem, and restart the replica SQL thread with "START REPLICA". We stopped at log 'binlog-bin.000026' position 573
2023-04-27T08:37:05.727187Z 133 [System] [MY-010597] [Repl] 'CHANGE REPLICATION SOURCE TO FOR CHANNEL 'group_replication_recovery' executed'. Previous state source_host='192.168.207.131', source_port= 3306, source_log_file='', source_log_pos= 4, source_bind=''. New state source_host='192.168.207.131', source_port= 3306, source_log_file='', source_log_pos= 4, source_bind=''.
2023-04-27T08:37:05.772920Z 212 [Warning] [MY-010897] [Repl] Storing MySQL user name or password information in the connection metadata repository is not secure and is therefore not recommended. Please consider using the USER and PASSWORD connection options for START REPLICA; see the 'START REPLICA Syntax' in the MySQL Manual for more information.
2023-04-27T08:37:05.783971Z 212 [System] [MY-014002] [Repl] Replica receiver thread for channel 'group_replication_recovery': connected to source 'repl@192.168.207.131:3306' with server_uuid=bed33e2c-c883-11ed-b4ea-de4fc9577006, server_id=6. Starting GTID-based replication.
2023-04-27T08:37:05.825297Z 214 [ERROR] [MY-011542] [Repl] Plugin group_replication reported: 'Table jj does not have any PRIMARY KEY. This is not compatible with Group Replication.'
2023-04-27T08:37:05.825745Z 214 [ERROR] [MY-010584] [Repl] Replica SQL for channel 'group_replication_recovery': Worker 1 failed executing transaction 'bed33e2c-c883-11ed-b4ea-de4fc9577006:3' at source log binlog-bin.000026, end_log_pos 793; The table in transaction bed33e2c-c883-11ed-b4ea-de4fc9577006:3 does not comply with the requirements by an external plugin. Error_code: MY-003635
2023-04-27T08:37:05.827415Z 213 [ERROR] [MY-010586] [Repl] Error running query, replica SQL thread aborted. Fix the problem, and restart the replica SQL thread with "START REPLICA". We stopped at log 'binlog-bin.000026' position 573
2023-04-27T08:37:31.919934Z 114 [System] [MY-011650] [Repl] Plugin group_replication reported: 'Plugin 'group_replication' is stopping.'
2023-04-27T08:37:35.179358Z 0 [System] [MY-011504] [Repl] Plugin group_replication reported: 'Group membership changed: This member has left the group.'
2023-04-27T08:37:35.573335Z 133 [System] [MY-010597] [Repl] 'CHANGE REPLICATION SOURCE TO FOR CHANNEL 'group_replication_recovery' executed'. Previous state source_host='192.168.207.131', source_port= 3306, source_log_file='', source_log_pos= 4, source_bind=''. New state source_host='192.168.207.131', source_port= 3306, source_log_file='', source_log_pos= 4, source_bind=''.
2023-04-27T08:37:35.885170Z 133 [System] [MY-010597] [Repl] 'CHANGE REPLICATION SOURCE TO FOR CHANNEL 'group_replication_recovery' executed'. Previous state source_host='192.168.207.131', source_port= 3306, source_log_file='', source_log_pos= 4, source_bind=''. New state source_host='<NULL>', source_port= 0, source_log_file='', source_log_pos= 4, source_bind=''.
2023-04-27T08:37:35.981738Z 114 [System] [MY-011565] [Repl] Plugin group_replication reported: 'Setting super_read_only=ON.'
2023-04-27T08:37:35.981855Z 114 [System] [MY-011651] [Repl] Plugin group_replication reported: 'Plugin 'group_replication' has been stopped.'
2023-04-27T08:49:49.480477Z 114 [System] [MY-010916] [Server] @@GLOBAL.GTID_PURGED was changed from '' to 'aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaa:1-39651:1000012-1000023:2000019, bed33e2c-c883-11ed-b4ea-de4fc9577006:1-3'.
2023-04-27T08:49:49.480612Z 114 [System] [MY-010917] [Server] @@GLOBAL.GTID_EXECUTED was changed from '' to 'aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaa:1-39651:1000012-1000023:2000019, bed33e2c-c883-11ed-b4ea-de4fc9577006:1-3'.
2023-04-27T08:49:59.932308Z 114 [System] [MY-013587] [Repl] Plugin group_replication reported: 'Plugin 'group_replication' is starting.'
2023-04-27T08:49:59.933161Z 114 [System] [MY-011565] [Repl] Plugin group_replication reported: 'Setting super_read_only=ON.'
2023-04-27T08:49:59.984812Z 476 [System] [MY-010597] [Repl] 'CHANGE REPLICATION SOURCE TO FOR CHANNEL 'group_replication_applier' executed'. Previous state source_host='<NULL>', source_port= 0, source_log_file='', source_log_pos= 4, source_bind=''. New state source_host='<NULL>', source_port= 0, source_log_file='', source_log_pos= 4, source_bind=''.
2023-04-27T08:50:01.196510Z 114 [System] [MY-011511] [Repl] Plugin group_replication reported: 'This server is working as secondary member with primary member address 192.168.207.131:3306.'
2023-04-27T08:50:01.197031Z 0 [System] [MY-011565] [Repl] Plugin group_replication reported: 'Setting super_read_only=ON.'
2023-04-27T08:50:01.197150Z 0 [System] [MY-013471] [Repl] Plugin group_replication reported: 'Distributed recovery will transfer data using: Incremental recovery from a group donor'
2023-04-27T08:50:01.204154Z 0 [System] [MY-011503] [Repl] Plugin group_replication reported: 'Group membership changed to 192.168.207.132:3306, 192.168.207.131:3306 on view 16825844038951171:4.'
2023-04-27T08:50:01.262231Z 489 [System] [MY-010597] [Repl] 'CHANGE REPLICATION SOURCE TO FOR CHANNEL 'group_replication_recovery' executed'. Previous state source_host='<NULL>', source_port= 0, source_log_file='', source_log_pos= 4, source_bind=''. New state source_host='192.168.207.131', source_port= 3306, source_log_file='', source_log_pos= 4, source_bind=''.
2023-04-27T08:50:01.302674Z 490 [Warning] [MY-010897] [Repl] Storing MySQL user name or password information in the connection metadata repository is not secure and is therefore not recommended. Please consider using the USER and PASSWORD connection options for START REPLICA; see the 'START REPLICA Syntax' in the MySQL Manual for more information.
2023-04-27T08:50:01.315440Z 490 [System] [MY-014002] [Repl] Replica receiver thread for channel 'group_replication_recovery': connected to source 'repl@192.168.207.131:3306' with server_uuid=bed33e2c-c883-11ed-b4ea-de4fc9577006, server_id=6. Starting GTID-based replication.
2023-04-27T08:50:01.455252Z 489 [System] [MY-010597] [Repl] 'CHANGE REPLICATION SOURCE TO FOR CHANNEL 'group_replication_recovery' executed'. Previous state source_host='192.168.207.131', source_port= 3306, source_log_file='', source_log_pos= 4, source_bind=''. New state source_host='<NULL>', source_port= 0, source_log_file='', source_log_pos= 4, source_bind=''.
2023-04-27T08:50:01.563190Z 0 [System] [MY-011490] [Repl] Plugin group_replication reported: 'This server was declared online within the replication group.'
2023-04-27T08:50:02.196697Z 114 [System] [MY-014010] [Repl] Plugin group_replication reported: 'Plugin 'group_replication' has been started.'
最终处理:通过clone的方式恢复,确保集群恢复正常。
mysql> select * from performance_schema.replication_group_members;
+---------------------------+--------------------------------------+-----------------+-------------+--------------+-------------+----------------+----------------------------+
| CHANNEL_NAME | MEMBER_ID | MEMBER_HOST | MEMBER_PORT | MEMBER_STATE | MEMBER_ROLE | MEMBER_VERSION | MEMBER_COMMUNICATION_STACK |
+---------------------------+--------------------------------------+-----------------+-------------+--------------+-------------+----------------+----------------------------+
| group_replication_applier | 37d95ab1-df20-11ed-9771-f25d7ebfe251 | 192.168.207.137 | 3306 | RECOVERING | SECONDARY | 8.0.33 | XCom |
| group_replication_applier | b893257d-c885-11ed-8be0-6a62c8446dfa | 192.168.207.132 | 3306 | ONLINE | SECONDARY | 8.0.33 | XCom |
| group_replication_applier | bed33e2c-c883-11ed-b4ea-de4fc9577006 | 192.168.207.131 | 3306 | ONLINE | PRIMARY | 8.0.33 | XCom |
+---------------------------+--------------------------------------+-----------------+-------------+--------------+-------------+----------------+----------------------------+
3 rows in set (0.00 sec)
mysql> stop group_replication;
Query OK, 0 rows affected (3.43 sec)
mysql> SET GLOBAL clone_valid_donor_list = '192.168.207.131:3306';
Query OK, 0 rows affected (0.01 sec)
mysql> CLONE INSTANCE FROM 'root'@'192.168.207.131':3306 IDENTIFIED BY 'xxxxxx';
ERROR 1290 (HY000): The MySQL server is running with the --super-read-only option so it cannot execute this statement
mysql> show variables like 'super_read_only';
+-----------------------------------------+---------------------------+
| Variable_name | Value |
+-----------------------------------------+---------------------------+
| super_read_only | ON |
1 rows in set (0.01 sec)
mysql> set global super_read_only=off;
Query OK, 0 rows affected (0.00 sec)
mysql> CLONE INSTANCE FROM 'root'@'192.168.207.131':3306 IDENTIFIED BY '1qazXSW@';
Query OK, 0 rows affected (9.10 sec)
克隆完成后自动恢复集群同步状态:
mysql> show variables like '%super_read_only%';
ERROR 2013 (HY000): Lost connection to MySQL server during query
No connection. Trying to reconnect...
Connection id: 31
Current database: *** NONE ***
+-----------------------------------------+---------------------------+
| Variable_name | Value |
+-----------------------------------------+---------------------------+
| super_read_only | ON |
+-----------------------------------------+---------------------------+
1 rows in set (0.04 sec)
mysql> select * from performance_schema.replication_group_members;
+---------------------------+--------------------------------------+-----------------+-------------+--------------+-------------+----------------+----------------------------+
| CHANNEL_NAME | MEMBER_ID | MEMBER_HOST | MEMBER_PORT | MEMBER_STATE | MEMBER_ROLE | MEMBER_VERSION | MEMBER_COMMUNICATION_STACK |
+---------------------------+--------------------------------------+-----------------+-------------+--------------+-------------+----------------+----------------------------+
| group_replication_applier | 37d95ab1-df20-11ed-9771-f25d7ebfe251 | 192.168.207.137 | 3306 | ONLINE | SECONDARY | 8.0.33 | XCom |
| group_replication_applier | b893257d-c885-11ed-8be0-6a62c8446dfa | 192.168.207.132 | 3306 | ONLINE | SECONDARY | 8.0.33 | XCom |
| group_replication_applier | bed33e2c-c883-11ed-b4ea-de4fc9577006 | 192.168.207.131 | 3306 | ONLINE | PRIMARY | 8.0.33 | XCom |
+---------------------------+--------------------------------------+-----------------+-------------+--------------+-------------+----------------+----------------------------+
3 rows in set (0.00 sec)
mysql> show variables like '%clone_valid_donor_list';
+------------------------+-------+
| Variable_name | Value |
+------------------------+-------+
| clone_valid_donor_list | |
+------------------------+-------+
1 row in set (0.01 sec)
mysql>
clone克隆的日志显示如下:
2023-04-27T08:51:04.616793Z 114 [System] [MY-011650] [Repl] Plugin group_replication reported: 'Plugin 'group_replication' is stopping.'
2023-04-27T08:51:07.723636Z 0 [System] [MY-011504] [Repl] Plugin group_replication reported: 'Group membership changed: This member has left the group.'
2023-04-27T08:51:07.726169Z 114 [System] [MY-011565] [Repl] Plugin group_replication reported: 'Setting super_read_only=ON.'
2023-04-27T08:51:07.726244Z 114 [System] [MY-011651] [Repl] Plugin group_replication reported: 'Plugin 'group_replication' has been stopped.'
2023-04-27T08:52:08.966794Z 114 [Warning] [MY-013460] [InnoDB] Clone removing all user data for provisioning: Started
2023-04-27T08:53:30.983007Z 114 [Warning] [MY-013460] [InnoDB] Clone removing all user data for provisioning: Started
2023-04-27T08:53:32.053999Z 114 [Warning] [MY-013460] [InnoDB] Clone removing all user data for provisioning: Finished
2023-04-27T08:53:44.449273Z 0 [System] [MY-010910] [Server] /usr/sbin/mysqld: Shutdown complete (mysqld 8.0.33) MySQL Community Server - GPL.
2023-04-27T08:53:48.216892Z 0 [Warning] [MY-011068] [Server] The syntax 'log_slave_updates' is deprecated and will be removed in a future release. Please use log_replica_updates instead.
2023-04-27T08:53:48.216923Z 0 [Warning] [MY-011069] [Server] The syntax '--master-info-repository' is deprecated and will be removed in a future release.
2023-04-27T08:53:48.216936Z 0 [Warning] [MY-011069] [Server] The syntax '--relay-log-info-repository' is deprecated and will be removed in a future release.
2023-04-27T08:53:48.216984Z 0 [Warning] [MY-011069] [Server] The syntax '--transaction-write-set-extraction' is deprecated and will be removed in a future release.
2023-04-27T08:53:48.220001Z 0 [System] [MY-010116] [Server] /usr/sbin/mysqld (mysqld 8.0.33) starting as process 9952
2023-04-27T08:53:48.254492Z 1 [System] [MY-013576] [InnoDB] InnoDB initialization has started.
2023-04-27T08:53:49.967514Z 1 [System] [MY-013577] [InnoDB] InnoDB initialization has ended.
2023-04-27T08:53:50.556571Z 0 [System] [MY-013587] [Repl] Plugin group_replication reported: 'Plugin 'group_replication' is starting.'
2023-04-27T08:53:50.556832Z 0 [System] [MY-014010] [Repl] Plugin group_replication reported: 'Plugin 'group_replication' has been started.'
2023-04-27T08:53:50.795503Z 0 [Warning] [MY-010068] [Server] CA certificate ca.pem is self signed.
2023-04-27T08:53:50.795604Z 0 [System] [MY-013602] [Server] Channel mysql_main configured to support TLS. Encrypted connections are now supported for this channel.
2023-04-27T08:53:50.812020Z 0 [Warning] [MY-013595] [Server] Failed to initialize TLS for channel: mysql_admin. See below for the description of exact issue.
2023-04-27T08:53:50.812113Z 0 [Warning] [MY-010069] [Server] Failed to set up SSL because of the following SSL library error: SSL context is not usable without certificate and private key
2023-04-27T08:53:50.812136Z 0 [System] [MY-013603] [Server] No TLS configuration was given for channel mysql_admin; re-using TLS configuration of channel mysql_main.
2023-04-27T08:53:50.855325Z 0 [Warning] [MY-010604] [Repl] Neither --relay-log nor --relay-log-index were used; so replication may break when this MySQL server acts as a replica and has his hostname changed!! Please use '--relay-log=pg-1-relay-bin' to avoid this problem.
2023-04-27T08:53:51.045237Z 0 [System] [MY-010931] [Server] /usr/sbin/mysqld: ready for connections. Version: '8.0.33' socket: '/var/lib/mysql/mysql.sock' port: 3306 MySQL Community Server - GPL.
2023-04-27T08:53:51.045283Z 0 [System] [MY-013292] [Server] Admin interface ready for connections, address: '192.168.207.132' port: 33062
2023-04-27T08:53:51.045397Z 0 [System] [MY-011323] [Server] X Plugin ready for connections. Bind-address: '::' port: 33060, socket: /var/run/mysqld/mysqlx.sock
2023-04-27T08:53:51.046099Z 4 [System] [MY-011565] [Repl] Plugin group_replication reported: 'Setting super_read_only=ON.'
2023-04-27T08:53:51.078324Z 12 [System] [MY-010597] [Repl] 'CHANGE REPLICATION SOURCE TO FOR CHANNEL 'group_replication_applier' executed'. Previous state source_host='<NULL>', source_port= 0, source_log_file='', source_log_pos= 4, source_bind=''. New state source_host='<NULL>', source_port= 0, source_log_file='', source_log_pos= 4, source_bind=''.
2023-04-27T08:53:53.786402Z 4 [System] [MY-011511] [Repl] Plugin group_replication reported: 'This server is working as secondary member with primary member address 192.168.207.131:3306.'
2023-04-27T08:53:53.786885Z 0 [System] [MY-011565] [Repl] Plugin group_replication reported: 'Setting super_read_only=ON.'
2023-04-27T08:53:53.786998Z 0 [System] [MY-013471] [Repl] Plugin group_replication reported: 'Distributed recovery will transfer data using: Incremental recovery from a group donor'
2023-04-27T08:53:53.787602Z 0 [System] [MY-011503] [Repl] Plugin group_replication reported: 'Group membership changed to 192.168.207.132:3306, 192.168.207.131:3306 on view 16825844038951171:6.'
2023-04-27T08:53:53.843760Z 25 [System] [MY-010597] [Repl] 'CHANGE REPLICATION SOURCE TO FOR CHANNEL 'group_replication_recovery' executed'. Previous state source_host='<NULL>', source_port= 0, source_log_file='', source_log_pos= 4, source_bind=''. New state source_host='192.168.207.131', source_port= 3306, source_log_file='', source_log_pos= 4, source_bind=''.
2023-04-27T08:53:53.898027Z 26 [Warning] [MY-010897] [Repl] Storing MySQL user name or password information in the connection metadata repository is not secure and is therefore not recommended. Please consider using the USER and PASSWORD connection options for START REPLICA; see the 'START REPLICA Syntax' in the MySQL Manual for more information.
2023-04-27T08:53:53.938642Z 26 [System] [MY-014002] [Repl] Replica receiver thread for channel 'group_replication_recovery': connected to source 'repl@192.168.207.131:3306' with server_uuid=bed33e2c-c883-11ed-b4ea-de4fc9577006, server_id=6. Starting GTID-based replication.
2023-04-27T08:53:54.234436Z 25 [System] [MY-010597] [Repl] 'CHANGE REPLICATION SOURCE TO FOR CHANNEL 'group_replication_recovery' executed'. Previous state source_host='192.168.207.131', source_port= 3306, source_log_file='', source_log_pos= 4, source_bind=''. New state source_host='<NULL>', source_port= 0, source_log_file='', source_log_pos= 4, source_bind=''.
2023-04-27T08:53:54.293321Z 0 [System] [MY-011490] [Repl] Plugin group_replication reported: 'This server was declared online within the replication group.'
2023-04-27T08:56:26.102172Z 0 [System] [MY-011503] [Repl] Plugin group_replication reported: 'Group membership changed to 192.168.207.137:3306, 192.168.207.132:3306, 192.168.207.131:3306 on view 16825844038951171:7.'
2023-04-27T08:56:26.997107Z 87 [Warning] [MY-011809] [Server] Cannot replicate to server with server_uuid='37d95ab1-df20-11ed-9771-f25d7ebfe251' because the present server has purged required binary logs. The connecting server needs to replicate the missing transactions from elsewhere, or be replaced by a new server created from a more recent backup. To prevent this error in the future, consider increasing the binary log expiration period on the present server. The missing transactions are 'aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaa:39649-39652, bed33e2c-c883-11ed-b4ea-de4fc9577006:3'.
2023-04-27T08:57:27.134050Z 108 [Warning] [MY-011809] [Server] Cannot replicate to server with server_uuid='37d95ab1-df20-11ed-9771-f25d7ebfe251' because the present server has purged required binary logs. The connecting server needs to replicate the missing transactions from elsewhere, or be replaced by a new server created from a more recent backup. To prevent this error in the future, consider increasing the binary log expiration period on the present server. The missing transactions are 'aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaa:39649-39652, bed33e2c-c883-11ed-b4ea-de4fc9577006:3'.
2023-04-27T08:58:27.578643Z 129 [Warning] [MY-011809] [Server] Cannot replicate to server with server_uuid='37d95ab1-df20-11ed-9771-f25d7ebfe251' because the present server has purged required binary logs. The connecting server needs to replicate the missing transactions from elsewhere, or be replaced by a new server created from a more recent backup. To prevent this error in the future, consider increasing the binary log expiration period on the present server. The missing transactions are 'aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaa:39649-39652, bed33e2c-c883-11ed-b4ea-de4fc9577006:3'.
2023-04-27T08:58:34.120195Z 0 [Warning] [MY-011499] [Repl] Plugin group_replication reported: 'Members removed from the group: 192.168.207.137:3306'
2023-04-27T08:58:34.120335Z 0 [System] [MY-011503] [Repl] Plugin group_replication reported: 'Group membership changed to 192.168.207.132:3306, 192.168.207.131:3306 on view 16825844038951171:8.'
2023-04-27T09:00:52.610733Z 0 [System] [MY-011503] [Repl] Plugin group_replication reported: 'Group membership changed to 192.168.207.137:3306, 192.168.207.132:3306, 192.168.207.131:3306 on view 16825844038951171:9.'
2023-04-27T09:00:53.092635Z 0 [System] [MY-011492] [Repl] Plugin group_replication reported: 'The member with address 192.168.207.137:3306 was declared online within the replication group.'
后续改进建议:
1.mgr环境要求表必须建立主键,然后才能dml操作。
2.业务务必连接浮动IP,不要独立连接本节点,避免误操作导致的数据不一致,最终使节点加入集群失败。
配置浮动IP可参考https://www.modb.pro/db/624348
最后修改时间:2023-05-08 14:37:01
「喜欢这篇文章,您的关注和赞赏是给作者最好的鼓励」
关注作者
【版权声明】本文为墨天轮用户原创内容,转载时必须标注文章的来源(墨天轮),文章链接,文章作者等基本信息,否则作者和墨天轮有权追究责任。如果您发现墨天轮中有涉嫌抄袭或者侵权的内容,欢迎发送邮件至:contact@modb.pro进行举报,并提供相关证据,一经查实,墨天轮将立刻删除相关内容。




