KES在linux操作系统中的安装

一、走进Kingbase ES（简称KES）
1. 数据库基础常识
```
在 "Kingbase ES" 中，"ES" 是指 "Enterprise Edition"，即企业版。

1、数据（Data）
描述事务的符号记录称为数据，数据可经过数字化后存入计算机中。
2、数据库（Database，DB）
数据库是长期存储在计算机内，有组织、可共享的大量数据的集合；
库中数据按照一定数据模型组织、描述和存储，有较小的冗余度、较高的数据独立性和一扩展性，为各用户共享。
3、数据库管理系统（DBMS）
是一个可以科学地组织和存储数据，高效地获取和维护数据的系统软件；包括以下功能：
<1>.数据库的建立和维护功能
<2>.数据库定义和创建
<3>.提供DML语言对数据进行存取
<4>.数据组织、存储和管理
<5>.事务管理和运行管理
<6>.与网络中其他软件的通信功能，异构数据库之间的互访和互操作功能
<7>.其他扩展功能
4、数据库系统（DBS）
由DB、DBMS及其应用开发工具、APP（数据库管理员）组成的存储、管理、处理和维护数据的应用系统。
```
### 2. KES的技术优势
```
KES具有“三高”、“三易”的优势。

高可靠、高性能、高安全
易管理、易使用、易迁移
```
## 二、KES装卸和启停
### 1. 背景知识
```
Kingbase ES跟x86-64以及国产鲲鹏、龙芯、飞腾、兆芯、申威、等硬件平台做过适配，
适用操作系统：
Linux：Red Hat、CentOS、AIX、SUSE、Turbo、Solaris、HP-UX、Rocky OS、红旗、麒麟Kylin、UOS
Windows：全系列

以下演示使用：CentOS 7.6 和 KES V8R6
```
### 2. KES安装
```
1. linux参数配置
1）安装前执行optimize_system_conf.sh（内容见附录1）优化操作系统
作用：优化sysctl.conf中的内核参数、limits.conf中的资源限制参数，优化脚本还有创建kingbase操作系统用户并对其资源限制进行优化。
需要手动关闭selinux
2）optimize_database_conf.sh
作用：

2. 创建目录
[root@node01 ~]# mkdir -p /KingbaseES/V8 /backup /archive /data
[root@node01 ~]# chown -R kingbase:kingbase /KingbaseES
[root@node01 ~]# chown -R kingbase:kingbase /backup
[root@node01 ~]# chown -R kingbase:kingbase /archive
[root@node01 ~]# chown -R kingbase:kingbase /install
[root@node01 ~]# chown -R kingbase:kingbase /data
[root@node01 ~]# chmod -R 755 /KingbaseES
[root@node01 ~]# chmod -R 755 /backup
[root@node01 ~]# chmod -R 755 /archive
[root@node01 ~]# chmod -R 755 /install
[root@node01 ~]# chmod -R 700 /data

install 安装软件存储目录（仅用来存放安装文件，操作系统优化时创建）
backup 备份目录
archive 归档目录
data 数据存储目录
/KingbaseES/V8 KES安装目录

3. 上传安装盘iso、检查md5、挂载
3.1 使用root账户将iso文件上传到/install目录
3.2 检查md5值
[root@node01 install]# md5sum KingbaseES_V008R006C005B0023_Lin64_single_install.iso
0a135a2451529ef62fd1be828fea56d4 KingbaseES_V008R006C005B0023_Lin64_single_install.iso
3.3 使用root用户挂载KES光盘文件
[root@node01 install]# mount -o loop KingbaseES_V008R006C005B0023_Lin64_single_install.iso /mnt
mount: /dev/loop0 写保护，将以只读方式挂载
3.4 使用kingbase用户复制挂载后的安装文件到/install目录
[kingbase@node01 ~]$ mkdir /install/KES-V8R6C5B23-INSTALL
[kingbase@node01 ~]$ cp -a /mnt/* /install/KES-V8R6C5B23-INSTALL

4. 上传授权文件
使用root用户将授权文件（license_12350_0.dat）上传到/install目录
#给/install目录重新修改所有者和所属组
[root@node01 ~]# chown -R kingbase:kingbase /install

5. 打开linux桌面终端
使用kingbase/kingbase用户登录
5.1 安装KES程序
[kingbase@node01 ~]$ cd /install/KES-V8R6C5B23-INSTALL/
[kingbase@node01 KES-V8R6C5B23-INSTALL]$ bash setup.sh
点击“下一步”，勾选“我接受许可协议条款”，点击“下一步”，选择“完全安装”，
选择授权文件，(有报错忽略)点击下一步，
你想安装在哪个位置（W）？ 选择：/KingbaseES/V8
点击“下一步”，“安装”
5.2 创建和初始化数据库集簇

1）请选择一个文件夹（F）：选择：/data
点击“下一步”

2）端口：54321 （可自定义）

3）账户：system
密码：（自定义）

4）Ecoding：（根据需求自定义）

5）DatabaseMode：
选择“oracle”即可，点击“下一步”，

6）将 KingbaseES 注册为系统服务
复制“/KingbaseES/V8/Scripts/root.sh”，使用root用户执行。

7）点“完成”。

6. 将KES服务注册为linux系统服务
使用root用户执行 /KingbaseES/V8/Scripts/root.sh
#重启服务器后查看kingbase8d服务的状态
[root@node01 ~]# systemctl status kingbase8d

7. 重启操作系统确认KES服务可自启动

8. 确认是否已安装
8.1 查看安装过程日志
若安装有问题会在/install/KES-V8R6C5B23-INSTALL产生错误日志（*.log）
8.2 查看开始菜单中是否已成功安装相关程序
8.3 查看相关进程是否启动
[kingbase@node01 ~]$ ps -ef |grep -v grep |grep -i kingbase
8.4 验证数据库连接是否正常
[kingbase@node01 ~]$ /KingbaseES/V8/Server/bin/ksql test system

9. 启动或停止KES服务的几种方法
9.1 root用户使用systemctl管理KES服务
[root@node01 ~]# systemctl start|stop|restart|status kingbase8d
9.2 root用户使用service管理KES服务
[root@node01 ~]# service kingbase8d start|stop|restart|status
9.3 kingbase用户使用金仓sys_ctl命令管理KES

10. 配置系统环境变量
cat /home/kingbase/.bashrc
export PATH=/KingbaseES/V8/Server/bin/:$PATH
export KINGBASE_DATA=/data
export KINGBASE_DATABASE=test
export KINGBASE_USER=system
```
### 3. KES卸载
```
1. 使用root用户执行/KingbaseES/V8/Scripts/rootuninstall.sh脚本移除KES开机自启服务

2. 卸载KES程序
方法1、使用kingbase用户登录到linux桌面，点击“卸载”进行卸载。
方法2、使用kingbase用户执行 bash /KingbaseES/V8/Uninstall/Uninstaller -i console 进行卸载
脚本执行完毕执行echo $? 若返回数值为0则表示执行成功。

3. 清除安装目录下的残留文件
方法1、执行rm -rf /KingbaseES/V8/直接删除安装目录
方法2、执行mv /KingbaseES/V8 /KingbaseES/V8.bak 将安装目录改名。
```
## 三、附录
### 3.1 附录1 - optimize_system_conf.sh
```
#!/bin/bash

####################################################################################################################
###
### Descipt: this script help us to make a base optimization for system
### Author : HM
### Create : 2020-04-28
###
### Usage :
### ./optimize_system_conf.sh
###
####################################################################################################################

echo "This tool help use to make a base optimization for system"
echo ""


#1.optimize System configuration
optimizeSystemConf(){
conf_exist=$(cat /etc/sysctl.conf|grep kingbase|wc -l)
if [ $conf_exist -eq 0 ]; then
echo "optimize system core conf"
cat >> /etc/sysctl.conf <<EOF
#add by kingbase
#/proc/sys/kernel/优化
# 10000 connect remain:
kernel.sem = 250 162500 250 650

#notice: shall shmmax is base on 16GB, you may adjust it for your MEM
#for 16GB Mem:
kernel.shmall = 3774873
kernel.shmmax = 8589934592

#for 32GB Mem:
#kernel.shmall = 7549747
#kernel.shmmax = 17179869184
#for 64GB Mem:
#kernel.shmall = 15099494
#kernel.shmmax = 34359738368
#for 128GB Mem:
#kernel.shmall = 30198988
#kernel.shmmax = 68719476736
#for 256GB Mem:
#kernel.shmall = 60397977
#kernel.shmmax = 137438953472
#for 512GB Mem:
#kernel.shmall = 120795955
#kernel.shmmax = 274877906944

kernel.shmmni = 4096

vm.dirty_background_ratio=2
vm.dirty_ratio = 40

vm.overcommit_memory = 2
vm.overcommit_ratio = 90

vm.swappiness = 0

fs.aio-max-nr = 1048576
fs.file-max = 6815744
fs.nr_open = 20480000

# TCP端口使用范围
net.ipv4.ip_local_port_range = 10000 65000
net.ipv4.tcp_keepalive_time = 1200
net.ipv4.tcp_keepalive_probes = 3
net.ipv4.tcp_keepalive_intvl = 30
net.ipv4.tcp_max_syn_backlog = 8192
net.ipv4.tcp_max_tw_buckets = 6000
# 记录的那些尚未收到客户端确认信息的连接请求的最大值
net.ipv4.tcp_max_syn_backlog = 65536
# 每个网络接口接收数据包的速率比内核处理这些包的速率快时，允许送到队列的数据包的最大数目
net.core.somaxconn=1024
net.core.netdev_max_backlog = 32768
net.core.wmem_default = 8388608
net.core.wmem_max = 1048576
net.core.rmem_default = 8388608
net.core.rmem_max = 16777216
net.ipv4.tcp_synack_retries = 2
net.ipv4.tcp_fin_timeout = 30
net.ipv4.tcp_syn_retries = 2
net.ipv4.route.gc_timeout = 100
net.ipv4.tcp_wmem = 8192 436600 873200
net.ipv4.tcp_rmem = 32768 436600 873200
net.ipv4.tcp_mem = 94500000 91500000 92700000
net.ipv4.tcp_max_orphans = 3276800

EOF
else
echo "system configuration is already optimized, so we do nothing"
fi
}


#2.Optimize Limit
optimizeLimitConf(){
conf_exist=$(cat /etc/security/limits.conf|grep kingbase|wc -l)
if [ $conf_exist -eq 0 ]; then
echo "optimize limit configuration"
cat >> /etc/security/limits.conf <<EOF
#add by kingbase
kingbase soft nproc 65536
kingbase hard nproc 65536
kingbase soft nofile 65536
kingbase hard nofile 65536
kingbase soft stack 10240
kingbase hard stack 32768
kingbase soft core unlimited
kingbase hard core unlimited
EOF

else
echo "limit is already optimized, so we do nothing"
fi

# modify nproc.conf
if [ -f /etc/security/limits.d/90-nproc.conf ]; then
conf_exist=$(cat /etc/security/limits.d/90-nproc.conf |grep kingbase|wc -l)
if [ $conf_exist -eq 0 ]
then
echo "90-nproc modifing"
cat >> /etc/security/limits.d/90-nproc.conf <<EOF
kingbase soft nproc 65536
EOF
else
echo "90-nproc already modified, so we do nothing"
fi
elif [ -f /etc/security/limits.d/20-nproc.conf ]; then
conf_exist=$(cat /etc/security/limits.d/20-nproc.conf |grep kingbase|wc -l)
if [ $conf_exist -eq 0 ]
then
echo "20-nproc modifing"
cat >> /etc/security/limits.d/20-nproc.conf <<EOF
kingbase soft nproc 65536
EOF
else
echo "20-nproc already modified, so we do nothing"
fi
fi
}

#3.optimize selinux
optimizeSelinux(){
if [ -f /etc/selinux/config ]; then
conf_exist=$(cat /etc/selinux/config|grep kingbase|wc -l)
if [ $conf_exist -eq 0 ]; then
echo "disable selinux"
cat >> /etc/selinux/config <<EOF
#kingbase add:
SELINUX=disabled
EOF
else
echo "selinux is already disabled, so wo do nothing"
fi
fi
}

#4.off firwalld -- this must user do it myself

#create user if not exists
create_kingbase_user_if_not_exist(){
#create group if not exists
egrep "^kingbase:" /etc/group >& /dev/null
if [ $? -ne 0 ]
then
groupadd kingbase
fi
egrep "^kingbase:" /etc/passwd >& /dev/null
if [ $? -ne 0 ]
then
useradd -m -g kingbase kingbase
echo kingbase|passwd --stdin kingbase >& /dev/null
if [ $? -ne 0 ]
then
echo "kingbase:kingbase" | chpasswd >& /dev/null
fi
fi
}

#main:

#create user if not exists
create_kingbase_user_if_not_exist

#1.optimize System configuration
echo "1.optimize system core configuration:"
optimizeSystemConf
sysctl -p >>/dev/null 2>&1
echo ""

#2.Optimize Limit
echo "2.optimize limit:"
optimizeLimitConf
echo ""
echo "3.check limit:"
su - kingbase -c 'ulimit -a'|grep 'open files'
echo ""

#3.optimize selinux
echo "4.disable selinux:"
optimizeSelinux
echo ""
#getenforce not support for UOS
#echo "5.check selinux:"
#getenforce
#echo ""

#4.off firwalld -- must do ti by user

#5.device scheduler -- optimize by kingbase DBA

#6.date must do it by user

#7.IPC, some system have this, so do it by user
#8.disable firwalld
echo "8.disable firewall"
systemctl stop firewalld.service
systemctl disable firewalld.service
echo "firewall is disabled."

```
### 3.2 附录2 - optimize_database_conf.sh
```
#!/bin/bash

####################################################################################################################
###
### Descipt: this script help us to make a base optimization for database
### Author : HM
### Create : 2020-04-28
###
### Usage :
### ./optimize_database_conf.sh
###
####################################################################################################################

echo "This tool help use to make a base optimization for database"
echo ""

set -e
#1. get database data first, if not, exit.
# TODO: or input the data dir?
check_database_data(){
main_proc_num=$(ps -ef|grep -E "kingbase:.*checkpointer"|grep -v grep|wc -l)
case $main_proc_num in
0)
echo "the database is not on live, please input kingbase home path:"
read kingbase_home
if [[ ! -d "$kingbase_home" ]] || [[ ! -f "$kingbase_home/Server/bin/kingbase" ]]; then
echo "kingbase home path is error, please check it and try again !"
echo ""
exit
fi
server_path=$kingbase_home/Server
bin_path=$server_path/bin
kingbase_path=$bin_path/kingbase

echo "can not get data path from main process, please input the data path:"
read data_dir
if [[ ! -d "$data_dir" ]] || [[ ! -f "$data_dir/kingbase.conf" ]]; then
echo "data path is error"
echo "you can use: \"find / -name kingbase.conf\" to find it"
echo "please check and try again !"
exit 10
fi
;;
1)
KESPID=$(ps -ef|grep -E "kingbase:.*checkpointer"|grep -v grep|awk '{print $3}'|head -1)
kingbase_path=$(ls -l /proc/$KESPID/exe|awk '{print $NF}')
data_dir=$(ls -l /proc/$KESPID/cwd|awk '{print $NF}')
bin_path=${kingbase_path%/*}
server_path=${bin_path%/*}
kingbase_home=${server_path%/*}
;;
*)
echo "There are multiple KingbaseES main processes, please select one: "
ps -ef|grep -E "kingbase:.*checkpointer"|grep -v grep|awk '{print $3}'|while read line; do echo "KES PID: $line"; ls -l /proc/$line/exe|awk '{print $NF}'|xargs echo " CMD : " ;ls -l /proc/$line/cwd|awk '{print $NF}'|xargs echo " DATA: " ; done
echo "Enter KingbaseES PID:"
read KESPID
kingbase_path=$(ls -l /proc/$KESPID/exe|awk '{print $NF}')
data_dir=$(ls -l /proc/$KESPID/cwd|awk '{print $NF}')
bin_path=${kingbase_path%/*}
server_path=${bin_path%/*}
kingbase_home=${server_path%/*}
esac

if [ ! -f "$kingbase_path" ]; then
echo "kingbase_path is error"
echo "please check and try again !"
exit 10
fi

echo
echo "kingbase_path : "$kingbase_path
echo "server_path : "$server_path
echo "kingbase_home : "$kingbase_home
echo "database data_dir: "$data_dir
}

#2. back kingbase.conf first
# TODO: we just modify the kingbase.conf or kingbase.auto.conf?
back_kingbase_conf(){
kingbase_conf_back="kingbase.conf_back_"$(date "+%Y-%m-%d_%H_%M_%S")
cp $data_dir/kingbase.conf $data_dir/$kingbase_conf_back
echo "before optimize the database, back kingbase.conf to " $kingbase_conf_back
}


#3. get system source conf, the optimize base info
get_system_config(){
#1) get cpu cores:
cpu_cores=$(cat /proc/cpuinfo |grep 'processor'|wc -l)
echo "system CPU cores: " $cpu_cores

#2) get memery KB:
mem_kb=$(cat /proc/meminfo |grep MemTotal|awk '{print $2}')
echo "system Mem: " $mem_kb "KB as " $(echo "$mem_kb/1024"|bc) "MB"


#3) db_data path:
#data_dir=$(ps -ef|grep kingbase|grep D|grep data|awk '{print $10}')
#echo "database data dir: " $data_dir

#4) get disk type:
# this kind conf optimize by kingbaser
# mount check data divice name for optimize
#is_ssd=$(cat /sys/block/$DIVIE_NAME/queue/rotational)
#1: SATA
#0: SSD
}

#4. optimize database memory configuration
#shared_buffers = 128MB
#effective_cache_size = 4GB
#maintenance_work_mem = 64MB
#wal_buffers = -1
#work_mem = 16MB
#min_wal_size = 80MB
#max_wal_size = 1GB
optimize_db_mem(){
kingbase_conf=$data_dir/kingbase.conf
shared_mem=$(echo "$mem_kb/1024/4"|bc)
echo "shared_mem: " $shared_mem "MB"

cat >>$kingbase_conf <<EOF
#add by optimize tool:
shared_buffers = ${shared_mem}MB
effective_cache_size = $(echo "$mem_kb/1024 - $shared_mem"|bc)MB
min_wal_size = 2GB
max_wal_size = 8GB
EOF
if [ $mem_kb -lt $(echo "32*1024*1024"|bc) ]; then
echo "maintenance_work_mem = $(echo "$shared_mem/4"|bc)MB">>$kingbase_conf
else
echo "maintenance_work_mem = 2GB">>$kingbase_conf
fi

#TODO: work_mem do not optimize rigth now
#TODO: temp_buffers do not optimize rigth now
}

##5. optimize database checkpoint
optimize_checkpoint(){
kingbase_conf=$data_dir/kingbase.conf
cat >>$kingbase_conf <<EOF
checkpoint_completion_target = 0.9
checkpoint_timeout = 30min
EOF

}

#6. optimize database parallel
optimize_parallel(){
kingbase_conf=$data_dir/kingbase.conf
echo "max_worker_processes = $cpu_cores">>$kingbase_conf
if [ $cpu_cores -ge 8 ]; then
echo "max_parallel_workers_per_gather = 4">>$kingbase_conf
elif [ $cpu_cores -ge 2 ]; then
echo "max_parallel_workers_per_gather = $(echo "$cpu_cores/2"|bc)">>$kingbase_conf
else
#do not open parallel
echo "do not open parallel"
fi
}

#7. restart database, make the conf work
restart_db(){
kingbase_path=$(ps -ef|grep kingbase|grep data|grep D|awk '{print $8}')
bin_path=${kingbase_path%/*}
su - kingbase -c "$bin_path/sys_ctl -D $data_dir restart -l restart.log"
}

#main:
echo "begin optimize database"
#1. get database data first, if not, exit.
echo "1.get database data, check database is alive"
check_database_data
echo ""

#2. back kingbase.conf first
echo "2.back kingbase.conf file"
back_kingbase_conf
echo ""

#3.get system conf
echo "3.get system resource"
get_system_config
echo ""

#4. optimize database memory configuration
echo "4.optimize database memory"
optimize_db_mem
echo ""

#5. optimize database checkpoint
echo "5.optimize database checkpoint"
optimize_checkpoint
echo ""

#6. optimize database parallel
echo "6.optimize database parallel"
optimize_parallel
echo ""

echo "end optimize database"
echo ""
echo "7.restart database to make those configuration work"
echo "please chose if restart database, 0: no, 1: yes:"
#7. restart database, make the conf work
read restart_option
if [ $restart_option -eq 1 ]; then
restart_db
else
echo "please restart database by hand to make those configuration work"
echo "usage:"
echo "su - kingbase -c "$bin_path/sys_ctl -D $data_dir restart -l restart.log""
fi

echo ""
echo "end"

```