暂无图片
暂无图片
暂无图片
暂无图片
暂无图片
首页 / 《OceanBase诊断系列》——诊断工具obdiag一键收集诊断信息实践

《OceanBase诊断系列》——诊断工具obdiag一键收集诊断信息实践

OceanBase社区传送门 2023-11-03
473

1. 前言

obdiag定位为OceanBase诊断工具。整体使用上备以下的特点：

  • 部署简单：提供rpm包和OBD上部署的模式，均可一键部署安装，可以选择部署到任意一台能连接到集群各个节点的上，并不局限于OBServer节点。
  • 开箱即用：使用中所依赖的python包，全部都是自包含的，只需要部署机器存在python2或者python3环境即可；
  • 集中式收集：单点部署，无需每台服务器部署。使用的时候只需要在部署机器上执行收集或分析命令即可；

obdiag 1.2.1版本支持：

  • 一键收集OB日志
  • 一键收集AWR报告
  • 一键收集主机信息
  • 一键收集OB堆栈信息
  • 一键收集(clog、slog解析后的日志)
  • 一键收集perf信息（扁鹊图、perf火焰图、pstack火焰图）
  • 一键收集并行SQL的执行详情信息
  • 一键收集OBPROXY的日志待解锁功能

obdiag 1.3.0版本支持：

  • 一键分析OB日志

2. 安装部署使用

2.1. OBD模式下安装使用obdiag

如果你的OceanBase集群是通过OBD安装部署的，并且OBD的版本大于2.1.0那么，你可以直接通过下面的方式使用。

2.1.1. 安装

  1. 安装OBD 2.1.0 及以上版本
sudo yum install -y yum-utils
sudo yum-config-manager --add-repo https://mirrors.aliyun.com/oceanbase/OceanBase.repo
sudo yum install -y ob-deploy
source /etc/profile.d/obd.sh
  1. 安装obdiag工具

使用该命令可部署obdiag工具可在本机安装部署obdiag, (如果用户不安装直接使用也会走自动安装流程)

obd obdiag deploy

2.1.2. 使用

2.1.2.1. obd obdiag gather

使用该命令可调用obdiag工具进行OceanBase相关的诊断信息收集

obd obdiag gather <gather type> <deploy name> [options]

gather type包含：

  • log：一键收集所属OceanBase集群的日志
  • sysstat：一键收集所属OceanBase集群主机信息
  • clog：一键收集所属OceanBase集群(clog日志)
  • slog：一键收集所属OceanBase集群(slog日志)
  • plan_monitor：一键收集所属OceanBase集群指定trace_id的并行SQL的执行详情信息
  • perf：一键收集所属OceanBase集群的perf信息（扁鹊图、perf火焰图、pstack火焰图）
  • obproxy_log：一键收集所属OceanBase集群所依赖的obproxy组件的日志
  • all：一键统一收集所属OceanBase集群的诊断信息，包括收集OceanBase日志/主机信息/OceanBase堆栈信息/OceanBase clog、slog日志/OceanBase perf信息（扁鹊图、perf火焰图、pstack火焰图)

2.2. 独立安装使用obdiag

2.2.1. obdiag下载

obdiag工具可从OceanBase官网下载免费下载，下载链接

2.2.2. obdiag下载

使用文档参见链接

3. OBD模式下使用体验

1. 部署安装

# 安装OBD
sudo yum install -y yum-utils
sudo yum-config-manager --add-repo https://mirrors.aliyun.com/oceanbase/OceanBase.repo
sudo yum install -y ob-deploy
source /etc/profile.d/obd.sh

# 安装部署OB集群
obd cluster deploy testobdiag -c config.yaml
# 启动OB集群
obd cluster start testobdiag

# 部署obdiag
obd obdiag deploy

1698980582

2. 收集最近半小时内的OB集群日志

$obd obdiag gather log testobdiag --since 30m
Get local repositories and plugins ok
obdiag plugin : oceanbase-diagnostic-tool-install-1.0
No updates detected. obdiag is already up to date.
Open ssh connection ok

...

ZipFileInfo:
+----------------+-----------+
| Node           | LogSize   |
+================+===========+
| 11.162.218.126 | 12.231M   |
+----------------+-----------+

...

Gather Ob Log Summary:
+----------------+-----------+---------+--------+-----------------------------------------------------------------------------------------------------------+
| Node           | Status    | Size    | Time   | PackPath                                                                                                  |
+================+===========+=========+========+===========================================================================================================+
| 11.162.218.126 | Completed | 12.231M | 5 s    | /home/test/gather_pack_20231103110502/ob_log_192.168.1.111_20231103103500_20231103110600.zip.             |
+----------------+-----------+---------+--------+-----------------------------------------------------------------------------------------------------------+
If you want to view detailed obdiag logs, please run:'obdiag display-trace --trace_id c5ca9bce-0279-3977-bb52-3ae0b35cd3a8'

Trace ID: c5f1e526-79f5-11ee-81e3-00163e01c7ce
If you want to view detailed obd logs, please run: obd display-trace c5f1e526-79f5-11ee-81e3-00163e01c7ce
$obd obdiag gather all testobdiag
Get local repositories and plugins ok
obdiag plugin : oceanbase-diagnostic-tool-install-1.0
No updates detected. obdiag is already up to date.
Open ssh connection ok

...

ZipFileInfo:
+----------------+-----------+
| Node           | LogSize   |
+================+===========+
| 11.162.218.126 | 17.469M   |
+----------------+-----------+

...

Gather Ob Log Summary:
+----------------+-----------+---------+--------+-----------------------------------------------------------------------------------------------------------+
| Node           | Status    | Size    | Time   | PackPath                                                                                                  |
+================+===========+=========+========+===========================================================================================================+
| 11.162.218.126 | Completed | 17.469M | 6 s    | /home/test/gather_pack_20231103111342/ob_log_192.168.1.111_20231103104340_20231103111440.zip              |
+----------------+-----------+---------+--------+-----------------------------------------------------------------------------------------------------------+

... 

Gather Sysstat Summary:
+----------------+-----------+----------+--------+---------------------------------------------------------------------------------------------+
| Node           | Status    | Size     | Time   | PackPath                                                                                    |
+================+===========+==========+========+=============================================================================================+
| 11.162.218.126 | Completed | 865.705K | 2 s    | /home/test/gather_pack_20231103111342/sysstat_192.168.1.111_20231103111348.zip              |
+----------------+-----------+----------+--------+---------------------------------------------------------------------------------------------+

...

Gather Perf Summary:
+----------------+-----------+---------+--------+------------------------------------------------------------------------------------------+
| Node           | Status    | Size    | Time   | PackPath                                                                                 |
+================+===========+=========+========+==========================================================================================+
| 11.162.218.126 | Completed | 12.530K | 6 s    | /home/test/gather_pack_20231103111342/perf_192.168.1.111_20231103111351.zip              |
+----------------+-----------+---------+--------+------------------------------------------------------------------------------------------+

...

Gather Ob stack Summary:
+----------------+-----------+---------+--------+----------------------------------------------------------------------------------------------+
| Node           | Status    | Size    | Time   | PackPath                                                                                     |
+================+===========+=========+========+==============================================================================================+
| 11.162.218.126 | Completed | 11.567K | 5 s    | /home/test/gather_pack_20231103111342/obstack2_192.168.1.111_20231103111358.zip              |
+----------------+-----------+---------+--------+----------------------------------------------------------------------------------------------+

...

Gather clog Summary:
+----------------+--------------------------------------------+--------+--------+---------------------------------------------------+
| Node           | Status                                     | Size   | Time   | PackPath                                          |
+================+============================================+========+========+===================================================+
| 11.162.218.126 | Error:4.2.0.0 not support gather clog/slog | 0.000B | 0 s    | /home/test/gather_pack_20231103111342             |
+----------------+--------------------------------------------+--------+--------+---------------------------------------------------+

..

Gather slog Summary:
+----------------+--------------------------------------------+--------+--------+---------------------------------------------------+
| Node           | Status                                     | Size   | Time   | PackPath                                          |
+================+============================================+========+========+===================================================+
| 11.162.218.126 | Error:4.2.0.0 not support gather clog/slog | 0.000B | 0 s    | /home/test/gather_pack_20231103111342             |
+----------------+--------------------------------------------+--------+--------+---------------------------------------------------+
If you want to view detailed obdiag logs, please run:'obdiag display-trace --trace_id afb5d4ad-b723-38bc-a660-d050c94132a3'

Trace ID: fc37da72-79f6-11ee-a367-00163e01c7ce
If you want to view detailed obd logs, please run: obd display-trace fc37da72-79f6-11ee-a367-00163e01c7ce

4. obdiag独立使用

1. 安装obdiag

sudo yum install -y yum-utils
sudo yum-config-manager --add-repo https://mirrors.aliyun.com/oceanbase/OceanBase.repo
sudo yum install -y oceanbase-diagnostic-tool
source /usr/local/oceanbase-diagnostic-tool/init.sh

2. obdiag配置

vi /usr/local/oceanbase-diagnostic-tool/config/config.yml
配置一：这部分的配置是 obdiag 自身的一些配置，包括自身的日志输出和外置的变量等
OBDIAG:
  BASIC:
    config_backup_dir: /tmp/oceanbase-diagnostic-tool/conf # 配置文件快速生成的时候会覆写 config.yml，为了让用户可查到上一次的配置，每次执行配置更新的时候会保存一份
    file_number_limit: 20 # 收集 observer 的日志的时候单机限制回传的最大文件数量
    file_size_limit: 2G # 收集 observer 的日志的时候单机限制回传的最大文件大小
  LOGGER: # obdiag 本身的日志打印相关配置
    file_handler_log_level: DEBUG # 输出到日志文件中的日志级别
    log_dir: /tmp/oceanbase-diagnostic-tool/log # obdiag 日志文件路径
    log_filename: odt.log # obdiag 日志文件名
    log_level: INFO # obdiag 日志输出的级别
    mode: obdiag
    stdout_handler_log_level: DEBUG

# 配置二：属于 OCP 的配置，有 OCP 托管的集群可以配置上, 没有 OCP 的话直接不填即可
OCP:
  LOGIN: # OCP 白屏的登录信息
    password: xxxx
    url: http://xxx.xxx.xxx.xxx:xxxx
    user: xxx
  METADB:
    dbname: xxx
    ip: xxx.xxx.xxx.xxx
    password: xxxx
    port: xxx
    user: xxx

# 配置三：需要收集的集群连接信息，用于查看 gather plan_monitor 的时候使用，如不需要该命令可不填该配置
OBCLUSTER:
  cluster_name: xxx
  host: xxx.xxx.xxx.xxx
  password: xxx
  port: xxx
  user: xxx

# 配置四：收集的节点的登录信息
NODES:
- ip: 192.168.1.11
  password: xxx
  port: xxx
  private_key: ''
  user: xxx
- ip: 192.168.1.12
  password: xxx
  port: xxx
  private_key: ''
  user: xxx

3. 收集指定时间的OB日志

obdiag gather log --scope observer --from 2023-10-30 16:25:00 --to 2023-10-30 18:30:00 --grep STORAGE --encrypt true

...
ZipFileInfo:
+-------------------+-----------+
| Node              | LogSize   |
+===================+===========+
| xxx.xxx.xxx.xxx   | 36.184M   |
+-------------------+-----------+
...

ZipFileInfo:
+-------------------+-----------+
| Node              | LogSize   |
+===================+===========+
| xxx.xxx.xxx.xxx   | 44.176M   |
+-------------------+-----------+
...

Summary:
+-------------------+-----------+----------+------------------+--------+------------------------------------------------------------------------+
| Node              | Status    | Size     | Password         | Time   | PackPath                                                               |
+===================+===========+==========+==================+========+========================================================================+
| xxx.xxx.xxx.xxx   | Completed | 36.762M  | HYmVourcUyRNP8Om | 19 s   | gather_pack_20231101183246/result_xxx.xxx.xxx.xxx_20231101183247.zip   |
+-------------------+-----------+----------+------------------+--------+------------------------------------------------------------------------+
| xxx.xxx.xxx.xxx   | Completed | 638.200M | 1RicMaiLUUNfemnj | 718 s  | gather_pack_20231101183246/result_xxx.xxx.xxx.xxx_20231101183918.zip   |
+-------------------+-----------+----------+------------------+--------+------------------------------------------------------------------------+
oceanbase
「喜欢这篇文章，您的关注和赞赏是给作者最好的鼓励」
关注作者
【版权声明】本文为墨天轮用户原创内容，转载时必须标注文章的来源（墨天轮），文章链接，文章作者等基本信息，否则作者和墨天轮有权追究责任。如果您发现墨天轮中有涉嫌抄袭或者侵权的内容，欢迎发送邮件至：contact@modb.pro进行举报，并提供相关证据，一经查实，墨天轮将立刻删除相关内容。

评论