0. ENV
obd 1.1.1升级到1.2.1正常；
oceanbase-ce 3.1.1升级到3.1.2正常；
obproxy 3.1.0升级到3.2.0异常。
1. 问题现象
obproxy从3.1.0升级到3.2.0，升级过程异常“Connect to obproxy x”
[admin@obp1 ~]$ obd cluster upgrade product -c obproxy -V 3.2.0Get local repositories and plugins ok+---------------------------------------------------------------------------------------------------------+| Search obproxy 3.2.0 Result |+---------+---------+---------+--------+------------------------------------------------------------------+| name | version | release | arch | md5 |+---------+---------+---------+--------+------------------------------------------------------------------+| obproxy | 3.2.0 | 1.el7 | x86_64 | c2ad988860a0fe5c07cbe3755fab0e2c165cf7a771e6c1f11741cc7208bf13d3 |+---------+---------+---------+--------+------------------------------------------------------------------+Download obproxy-3.2.0-1.el7.x86_64.rpm (7.80 M): 100% [####################################################################################] Time: 0:00:05 1.38 MB/sinstall obproxy-3.2.0 for local okOpen ssh connection ok[WARN] No such upgrade_route plugin for obproxy-3.1.0[WARN] No such upgrade_check plugin for obproxy-3.1.0[WARN] No such upgrade_check plugin for obproxy-3.2.0+-----------------------------------------------------------------------------------------+| Packages Will Be Used |+---------+---------+---------+--------+------------------------------------------+-------+| name | version | release | arch | md5 | mark |+---------+---------+---------+--------+------------------------------------------+-------+| obproxy | 3.1.0 | 1.el7 | x86_64 | 0b17cf0459a3b53c5a2febb6572894d183154c64 | start || obproxy | 3.2.0 | 1.el7 | x86_64 | 8d5c6978f988935dc3da1dbec208914668dcf3b2 | dest |+---------+---------+---------+--------+------------------------------------------+-------+If you use a non-official release, we cannot guarantee a successful upgrade or technical support when you fail. Make sure that you want to use the above package to upgrade. [y/n]: yRemote obproxy-3.2.0-8d5c6978f988935dc3da1dbec208914668dcf3b2 repository install okRemote obproxy-3.2.0-8d5c6978f988935dc3da1dbec208914668dcf3b2 repository lib check okStop obproxy okStart obproxy okobproxy program health check okConnect to obproxy x #升级异常
2. 分析处理过程
1) 再次升级问题依旧
[admin@obp1 ~]$ obd cluster upgrade product -c obproxy -V 3.2.0Get local repositories and plugins okOpen ssh connection okRemote obproxy-3.2.0-8d5c6978f988935dc3da1dbec208914668dcf3b2 repository install okRemote obproxy-3.2.0-8d5c6978f988935dc3da1dbec208914668dcf3b2 repository lib check okStop obproxy okStart obproxy okobproxy program health check okConnect to obproxy x
2) 通过obproxy登录observer正常
[admin@obp1 ~]$ obclient -hobp1 -P2883 -uroot@sys#product oceanbase -c -A -p'mw)))pwd5w/m'Welcome to the OceanBase. Commands end with ; or \g.Your MySQL connection id is 1048577Server version: 5.6.25 OceanBase 3.1.2 (r10000392021123010-d4ace121deae5b81d8f0b40afbc4c02705b7fc1d) (Built Dec 30 2021 02:47:29)Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others.Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.MySQL [oceanbase]> show databases;+--------------------+| Database |+--------------------+| oceanbase || information_schema || mysql || SYS || LBACSYS || ORAAUDITOR || test |+--------------------+7 rows in set (0.034 sec)
3) 查看版本已升级成功-3.2.0 1
[admin@obp1 ~]$ /home/admin/obproxy/bin/obproxy -V/home/admin/obproxy/bin/obproxy -Vobproxy (OceanBase 3.2.0 1)REVISION: 58-local-b91a4006ba52e107d1a3edb03a50f3d14ef63cd8BUILD_TIME: Nov 3 2021 06:51:33BUILD_FLAGS: -g -O2 -D_OB_VERSION=1000 -D_NO_EXCEPTION -D__STDC_LIMIT_MACROS -D__STDC_CONSTANT_MACROS -DNDEBUG -D__USE_LARGEFILE64 -D_FILE_OFFSET_BITS=64 -D_LARGE_FILE -D_LARGEFILE_SOURCE -D_LARGEFILE64_SOURCE -Wall -Wextra -Wunused-parameter -Wformat -Wconversion -Wno-deprecated -Wno-invalid-offsetof -finline-functions -fno-strict-aliasing -mtune=core2 -Wno-psabi -Wno-sign-compare -DGCC_52 -DSUPPORT_SSE4_2 -DHAVE_SCHED_GETCPU -DHAVE_REALTIME_COARSE -DOB_HAVE_EVENTFD -DHAVE_FALLOCATE -WerrorCopyright (c) 2021 OceanBaseOceanBase Database Proxy(ODP) is licensed under Mulan PubL v2.You can use this software according to the terms and conditions of the Mulan PubL v2.You may obtain a copy of Mulan PubL v2 at:http://license.coscl.org.cn/MulanPubL-2.0THIS SOFTWARE IS PROVIDED ON AN "AS IS" BASIS, WITHOUT WARRANTIES OF ANY KIND,EITHER EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO NON-INFRINGEMENT,MERCHANTABILITY OR FIT FOR A PARTICULAR PURPOSE.See the Mulan PubL v2 for more details.
4) 查看obproxy日志
[admin@obp1 ~]$ view /home/admin/obproxy/log/obproxy.log...[2022-01-08 15:12:03.144176] INFO [PROXY.CS] ob_mysql_transact.cpp:4925 [25649][Y0-7F809327BB00] [lt=47] [dc=0] succ to global variable proxy_route_policy(policy=)[2022-01-08 15:12:13.154946] INFO [PROXY.CS] ob_mysql_transact.cpp:4925 [25649][Y0-7F809327BB00] [lt=5] [dc=0] succ to global variable proxy_route_policy(policy=)[2022-01-08 15:12:13.161347] INFO [PROXY.CS] ob_mysql_transact.cpp:4925 [25649][Y0-7F809327BB00] [lt=28] [dc=0] succ to global variable proxy_route_policy(policy=)[2022-01-08 15:12:13.164714] INFO [PROXY] ob_server_state_processor.cpp:1329 [25649][Y0-7F809327BB00] [lt=31] [dc=0] rs list maybe update by rs refresh task, current used dummy entry maybe old one. We need stop to update server list and dummy entry this time(cluster_name_=product, last_rs_list_hash=0, last_server_list_hash=17872844444216291577, server_list=[[0]{server:"192.168.80.242:2881", is_dup_replica:false, role:"LEADER", type:"FULL"}, [1]{server:"192.168.80.243:2881", is_dup_replica:false, role:"FOLLOWER", type:"FULL"}, [2]{server:"192.168.80.244:2881", is_dup_replica:false, role:"FOLLOWER", type:"FULL"}])[2022-01-08 15:12:13.164869] INFO [PROXY.CS] ob_mysql_transact.cpp:4925 [25649][Y0-7F809327BB00] [lt=64] [dc=0] succ to global variable proxy_route_policy(policy=)[2022-01-08 15:12:33.138624] INFO [PROXY.CS] ob_mysql_transact.cpp:4925 [25649][Y0-7F809327BB00] [lt=17] [dc=0] succ to global variable proxy_route_policy(policy=)[2022-01-08 15:12:33.143075] INFO [PROXY.CS] ob_mysql_transact.cpp:4925 [25649][Y0-7F809327BB00] [lt=29] [dc=0] succ to global variable proxy_route_policy(policy=)[2022-01-08 15:12:33.145376] INFO [PROXY] ob_server_state_processor.cpp:1373 [25649][Y0-7F809327BB00] [lt=30] [dc=0] ObServerStateRefreshCont, update sys tennant's __all_dummy succ((tmp_entry={{this:0x7f8093341830, ref_count:4, cr_version:0, cr_id:0, create_time_us:1641625953145365, last_valid_time_us:1641625953145264, last_access_time_us:1641625953145264, last_update_time_us:0, schema_version:0, tenant_version:0, time_for_expired:0, state:"AVAIL"}, this:0x7f8093341830, is_inited:true, is_dummy_entry:true, is_entry_from_rslist:false, is_empty_entry_allowed:false, is_need_force_flush:false, has_dup_replica:false, cr_id:0, name:{cluster_name:"product", tenant_name:"sys", database_name:"oceanbase", package_name:"", table_name:"__all_dummy"}, table_id:1099511627911, table_type:"UNKNOWN", part_num:1, replica_num:3, buf_len:32, buf_start:0x7f8093341930, tenant_servers:{this:0x7f8093344640, is_inited:true, server_count:3, replica_count:3, partition_count:1, next_partition_idx:0, server_array:0x7f808d77eb80, server_array_:[[0]{server:"192.168.80.244:2881", is_dup_replica:false, role:"FOLLOWER", type:"FULL"}, [1]{server:"192.168.80.242:2881", is_dup_replica:false, role:"FOLLOWER", type:"FULL"}, [2]{server:"192.168.80.243:2881", is_dup_replica:false, role:"FOLLOWER", type:"FULL"}]}})[2022-01-08 15:12:33.145628] INFO [PROXY.CS] ob_mysql_transact.cpp:4925 [25649][Y0-7F809327BB00] [lt=128] [dc=0] succ to global variable proxy_route_policy(policy=)[2022-01-08 15:12:45.245113] INFO [PROXY.CS] ob_mysql_transact.cpp:4925 [25649][Y0-7F809327BB00] [lt=21] [dc=0] succ to global variable proxy_route_policy(policy=)[2022-01-08 15:12:45.248016] INFO [PROXY.CS] ob_mysql_transact.cpp:4925 [25649][Y0-7F809327BB00] [lt=10] [dc=0] succ to global variable proxy_route_policy(policy=)[2022-01-08 15:12:45.249415] INFO [PROXY] ob_server_state_processor.cpp:1329 [25649][Y0-7F809327BB00] [lt=22] [dc=0] rs list maybe update by rs refresh task, current used dummy entry maybe old one. We need stop to update server list and dummy entry this time(cluster_name_=product, last_rs_list_hash=0, last_server_list_hash=17872844444216291577, server_list=[[0]{server:"192.168.80.242:2881", is_dup_replica:false, role:"LEADER", type:"FULL"}, [1]{server:"192.168.80.243:2881", is_dup_replica:false, role:"FOLLOWER", type:"FULL"}, [2]{server:"192.168.80.244:2881", is_dup_replica:false, role:"FOLLOWER", type:"FULL"}])[2022-01-08 15:12:45.249570] INFO [PROXY.CS] ob_mysql_transact.cpp:4925 [25649][Y0-7F809327BB00] [lt=62] [dc=0] succ to global variable proxy_route_policy(policy=)
5) 定位问题
从日志上来看，升级是完成了，不会影响到业务使用。
OBD报升级失败的原因是应该是OBD机器不能链接到obproxy的proxysys上。猜测原因有二：
A. proxysys禁用了非127.1的ip访问，导致obd所在的机器不能建链，这种情况下请先使用客户端链接到proxysys上后执行alter proxyconfig set skip_proxy_sys_private_check = true。
B. proxysys的密码与obd中存储的不一致，这种情况下请先修改proxysys密码与obd中存储的一致，完成升级后在使用edit-config修改密码后使用reload使其生效。
3. 进一步解决
3.1 修改skip_proxy_sys_private_check参数为true
1) 查看proxysys是否有密码
[admin@obp1 ~]$ cd obproxy/[admin@obp1 obproxy]$ strings etc/obproxy_config.bin | grep sysobserver_sys_password1=observer_sys_password=obproxy_sys_password=e3fd448c516073714189b57233c9cf428ccb1bedskip_proxy_sys_private_check=Falsesyslog_level=INFO
2) obproxy主机使用127.0.0.1登录proxysys
[admin@obp1 obproxy]$ obclient -h127.0.0.1 -P2883 -uroot@proxysys -p'proxysys'Welcome to the OceanBase. Commands end with ; or \g.Your MySQL connection id is 2621459Server version: 5.6.25Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others.Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.MySQL [(none)]>
3) 更改参数skip_proxy_sys_private_check为true
MySQL [(none)]> show proxyconfig like 'skip_proxy_sys_private_check';+------------------------------+-------+------------------------------+-------------+---------------+| name | value | info | need_reboot | visible_level |+------------------------------+-------+------------------------------+-------------+---------------+| skip_proxy_sys_private_check | False | skip_proxy_sys_private_check | false | USER |+------------------------------+-------+------------------------------+-------------+---------------+1 row in set (0.001 sec)MySQL [(none)]> alter proxyconfig set skip_proxy_sys_private_check = true;Query OK, 0 rows affected (0.001 sec)MySQL [(none)]> show proxyconfig like 'skip_proxy_sys_private_check';+------------------------------+-------+------------------------------+-------------+---------------+| name | value | info | need_reboot | visible_level |+------------------------------+-------+------------------------------+-------------+---------------+| skip_proxy_sys_private_check | True | skip_proxy_sys_private_check | false | USER |+------------------------------+-------+------------------------------+-------------+---------------+1 row in set (0.000 sec)
3.2 置空proxysys密码--貌似与此无关，可跳过该步骤，执行下述操作
在proxysys下执行alter proxyconfig set obproxy_sys_password = '';[admin@obp1 obproxy]$ obclient -h127.0.0.1 -P2883 -uroot@proxysys -p'proxysys'Welcome to the OceanBase. Commands end with ; or \g.Your MySQL connection id is 2621461Server version: 5.6.25Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others.Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.MySQL [(none)]> alter proxyconfig set obproxy_sys_password = '';Query OK, 0 rows affected (0.001 sec)
3.3 重新执行obproxy升级-OK
[admin@obp1 ~]$ obd cluster upgrade product -c obproxy -V 3.2.0Get local repositories and plugins okOpen ssh connection okRemote obproxy-3.2.0-8d5c6978f988935dc3da1dbec208914668dcf3b2 repository install okRemote obproxy-3.2.0-8d5c6978f988935dc3da1dbec208914668dcf3b2 repository lib check okStop obproxy okStart obproxy okobproxy program health check okConnect to obproxy ok+-------------------------------------------------+| obproxy |+---------------+------+-----------------+--------+| ip | port | prometheus_port | status |+---------------+------+-----------------+--------+| 192.168.80.245 | 2883 | 2884 | active |+---------------+------+-----------------+--------+
3.4 后续处理
为了避免后续出问题，还请通过edit-config设置下skip_proxy_sys_private_check：true，并reload。
1) 编辑集群配置文件
[admin@obp1 ~]$ obd cluster edit-config productSearch param plugin and load okParameter check okSave deploy "product" configurationUse `obd cluster reload product` to make changes take effect.在obproxy的global下，添加"skip_proxy_sys_private_check: true"。obproxy:servers:- 192.168.80.245global:skip_proxy_sys_private_check: truelisten_port: 2883 # External port. The default value is 2883.
2) 重新加载配置文件
[admin@obp1 ~]$ obd cluster edit-config product[admin@obp1 ~]$ obd cluster reload productGet local repositories and plugins okLoad cluster param plugin okOpen ssh connection okCluster status check okConnect to obproxy okConnect to observer okproduct reload
4. 小结
OceanBase早期example里没有写上skip_proxy_sys_private_check：true导致的问题。如果后续还obproxy从3.1.0开始的升级，请先检查下skip_proxy_sys_private_check：true是否有配置。没有的话需要建加上这个配置项并reload，再进行升级。
旨在交流，不足之处，还望抛砖。
作者:王坤，微信公众号:rundba，欢迎转载，转载请注明出处。
如需公众号转发，请联系wx: landnow。
文章转载自rundba，如果涉嫌侵权，请发送邮件至：contact@modb.pro进行举报，并提供相关证据，一经查实，墨天轮将立刻删除相关内容。
