OB4.X集群状态running但是oceanbase-ce is not running 

问题描述

查看集群中的租户时无租户反馈，警告oceanbase-ce is not running，但是查询集群状态明明是running

--查看集群中的租户
[root@localhost ~]# obd cluster tenant show dxjobsingle -g
Get local repositories ok
[WARN] 192.168.40.110 oceanbase-ce is not running 
See https://www.oceanbase.com/product/ob-deployer/error-codes .
Trace ID: 20060750-8ec0-11f0-91da-000c29e13c26
If you want to view detailed obd logs, please run: obd display-trace 20060750-8ec0-11f0-91da-000c29e13c26

问题原因

虚拟机关机再开机后未启动集群

obd 运维的数据库，启停需要使用obd进行操作，停机时候没有使用obd会导致集群状态running但是oceanbase-ce is not running。

分析过程

查看集群状态

dxjobsingle集群状态是ruuning

obd cluster list

输出如下：

[root@localhost ~]# obd cluster list
+----------------------------------------------------------------+
|                          Cluster List                          |
+-------------+--------------------------------+-----------------+
| Name        | Configuration Path             | Status (Cached) |
+-------------+--------------------------------+-----------------+
| dxjobsingle | /root/.obd/cluster/dxjobsingle | running         |
+-------------+--------------------------------+-----------------+
Trace ID: 5bc2bd22-8ebd-11f0-810e-000c29e13c26
If you want to view detailed obd logs, please run: obd display-trace 5bc2bd22-8ebd-11f0-810e-000c29e13c26

查看集群中的租户

无返回租户却返回了[WARN] 192.168.40.110 oceanbase-ce is not running

查看集群中的租户
[root@localhost ~]# obd cluster tenant show dxjobsingle -g
Get local repositories ok
[WARN] 192.168.40.110 oceanbase-ce is not running 
See https://www.oceanbase.com/product/ob-deployer/error-codes .
Trace ID: 20060750-8ec0-11f0-91da-000c29e13c26
If you want to view detailed obd logs, please run: obd display-trace 20060750-8ec0-11f0-91da-000c29e13c26

查看当前obd 内指定集群的状态

参数 deploy name 为部署名，可以理解为配置文件的别名。自 V3.2.0 起，若 obd 设置了加密密钥，执行 display 命令输出的集群信息中，密码相关内容为加密状态。若需输出明文密码，可在命令中添加 --epk/--encryption-passkey 指定密钥用于验证。

通过ob-configserver is not runing可推测出数据库集群未启动，集群启动前有以下工作：

1.组件检查顺序：

ob-configserver

obagent

prometheus

grafana

2.启动顺序：

ob-configserver

observer

obshell

obproxy

obagent

prometheus

grafana

[root@localhost ~]# obd cluster display dxjobsingle
Get local repositories and plugins ok
Open ssh connection ok
[WARN] 192.168.40.110 ob-configserver is not running
See https://www.oceanbase.com/product/ob-deployer/error-codes .
Trace ID: 0b992440-8ec1-11f0-a576-000c29e13c26
If you want to view detailed obd logs, please run: obd display-trace 0b992440-8ec1-11f0-a576-000c29e13c26

解决办法

启动集群后再次查看租户正常

启动集群

[root@localhost ~]# obd cluster start dxjobsingle -S
Get local repositories ok
Load cluster param plugin ok
Cluster status check ok
Check before start ob-configserver ok
Check before start obagent ok
Check before start prometheus ok
Check before start grafana ok
Start ob-configserver ok
ob-configserver program health check ok
cluster scenario: htap
Start observer ok
observer program health check ok
Connect to observer 192.168.40.110:2881 ok
obshell start ok
obshell program health check ok
start obproxy ok
obproxy program health check ok
Connect to obproxy ok
Start obagent ok
.......

查看租户

[root@localhost ~]# obd cluster tenant show dxjobsingle -g
Get local repositories ok
Get deployment connections ok
Connect to observer 192.168.40.110:2881 ok
Select tenant ok
+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
|                                                                                    tenant basic info                                                                                    |
+-------------+-------------+--------------------+--------------+---------+---------+-------------+---------------------+---------------------+---------------+-------------+-------------+
| tenant_name | tenant_type | compatibility_mode | primary_zone | max_cpu | min_cpu | memory_size | max_iops            | min_iops            | log_disk_size | iops_weight | tenant_role |
+-------------+-------------+--------------------+--------------+---------+---------+-------------+---------------------+---------------------+---------------+-------------+-------------+
| sys         | SYS         | MYSQL              | RANDOM       | 2.0     | 2.0     | 2G          | 9223372036854775807 | 9223372036854775807 | 2G            | 2           | PRIMARY     |
| dxj         | USER        | MYSQL              | RANDOM       | 1.0     | 1.0     | 2G          | 9223372036854775807 | 9223372036854775807 | 6G            | 1           | PRIMARY     |
+-------------+-------------+--------------------+--------------+---------+---------+-------------+---------------------+---------------------+---------------+-------------+-------------+
Trace ID: 9d04fd46-8ec1-11f0-be08-000c29e13c26
If you want to view detailed obd logs, please run: obd display-trace 9d04fd46-8ec1-11f0-be08-000c29e13c26