OpenTenBase集群搭建+初始化（2CN+3DN）

原创 helenli 2025-06-19

168

实验环境

机器：阿里云ECS实例，2 核（vCPU）8 GiB，Ubuntu 22.04

数量：共使用了4台，其中3台共同构成2CN+3DN的集群节点，另一台作为部署主机，用于编译源码、分发二进制包等。

原因：起初使用阿里云2核4GB服务器，虽然在源码编译时没有出现问题，但是在集群搭建时会报错分配内存失败，因此调整了内存配置为8GB。

一、OpenTenBase源代码编译

见上一篇博客

二、OpenTenBase集群搭建+初始化

2.1 准备工作

1 主机规划

目前4台Ubuntu 22.04主机，IP分别为：

172.18.178.78（用于编译源码、分发二进制包、执行部署）

172.18.178.77

172.18.178.76

172.18.178.73

2CN+3DN集群规划如下：

节点名称	IP	数据目录
GTM master	172.18.178.73	/data/opentenbase/data/gtm
GTM slave	172.18.178.76	/data/opentenbase/data/gtm
CN1	172.18.178.73	/data/opentenbase/data/coord
CN2	172.18.178.76	/data/opentenbase/data/coord
DN1 master	172.18.178.73	/data/opentenbase/data/dn001
DN1 slave	172.18.178.76	/data/opentenbase/data/dn001
DN2 master	172.18.178.76	/data/opentenbase/data/dn002
DN2 slave	172.18.178.77	/data/opentenbase/data/dn002
DN3 master	172.18.178.77	/data/opentenbase/data/dn003
DN3 slave	172.18.178.73	/data/opentenbase/data/dn003

2 系统配置

在所有节点上执行以下操作：

Bash
# 1. 创建opentenbase用户
sudo useradd -g opentenbase opentenbase
sudo passwd opentenbase # 设置密码

# 2. 创建数据目录
sudo mkdir -p /data/opentenbase/data
sudo chown -R opentenbase:opentenbase /data/opentenbase

# 3. 禁用SELinux (Ubuntu默认不安装)
# 4. 禁用防火墙或开放必要端口
sudo ufw disable

# 5. 安装依赖
sudo apt-get update
sudo apt-get install -y gcc make flex bison git libreadline-dev zlib1g-dev libssl-dev libxml2-dev libxslt-dev libperl-dev libicu-dev

3 配置SSH互信

在所有节点上以opentenbase用户执行，一路回车。

Bash
ssh-keygen -t rsa

然后将所有节点的公钥(id_rsa.pub)内容复制，在所有节点上执行以下命令：

Bash
ssh-copy-id -i ~/.ssh/id_rsa.pub opentenbase@172.18.178.73
ssh-copy-id -i ~/.ssh/id_rsa.pub opentenbase@172.18.178.76
ssh-copy-id -i ~/.ssh/id_rsa.pub opentenbase@172.18.178.77

验证：

ssh opentenbase@172.18.178.73 echo "OK"
ssh opentenbase@172.18.178.76 echo "OK"
ssh opentenbase@172.18.178.77 echo "OK"

以一台机器的截图为例：

2.2 选择172.18.178.78作为部署主机

在部署主机上执行源码编译，见上文。

2.3 配置集群

1 配置环境变量

在所有节点上，编辑~/.bashrc文件，添加：

Bash
export OPENTENBASE_HOME=/data/opentenbase/install/opentenbase_bin_v2.0
export PATH=$OPENTENBASE_HOME/bin:$PATH
export LD_LIBRARY_PATH=$OPENTENBASE_HOME/lib:$LD_LIBRARY_PATH
export PGDATA=/data/opentenbase/data

然后执行：

Bash
source ~/.bashrc

2 初始化pgxc_ctl.conf

在部署主机上(172.18.178.78)：

Shell
mkdir /data/opentenbase/pgxc_ctl
cd /data/opentenbase/pgxc_ctl
vim pgxc_ctl.conf

配置文件基于官方的双节点配置文件修改IP地址、添加第三个DN节点。文件内容如下：

Bash
#!/bin/bash
# Double Node Config

# change: IP
IP_1=172.18.178.73
IP_2=172.18.178.76
IP_3=172.18.178.77

pgxcInstallDir=/data/opentenbase/install/opentenbase_bin_v2.0
pgxcOwner=opentenbase
defaultDatabase=postgres
pgxcUser=$pgxcOwner
tmpDir=/tmp
localTmpDir=$tmpDir
configBackup=n
configBackupHost=pgxc-linker
configBackupDir=$HOME/pgxc
configBackupFile=pgxc_ctl.bak

#---- GTM ----------
gtmName=gtm
gtmMasterServer=$IP_1
gtmMasterPort=50001
gtmMasterDir=/data/opentenbase/data/gtm
gtmExtraConfig=none
gtmMasterSpecificExtraConfig=none
gtmSlave=y
gtmSlaveServer=$IP_2
gtmSlavePort=50001
gtmSlaveDir=/data/opentenbase/data/gtm
gtmSlaveSpecificExtraConfig=none

#---- Coordinators -------
coordMasterDir=/data/opentenbase/data/coord
coordArchLogDir=/data/opentenbase/data/coord_archlog

coordNames=(cn001 cn002 )
coordPorts=(30004 30004 )
poolerPorts=(31110 31110 )
coordPgHbaEntries=(0.0.0.0/0)
coordMasterServers=($IP_1 $IP_2)
coordMasterDirs=($coordMasterDir $coordMasterDir)
coordMaxWALsernder=2
coordMaxWALSenders=($coordMaxWALsernder $coordMaxWALsernder )
coordSlave=n
coordSlaveSync=n
coordArchLogDirs=($coordArchLogDir $coordArchLogDir)

coordExtraConfig=coordExtraConfig
cat > $coordExtraConfig <<EOF
#================================================
# Added to all the coordinator postgresql.conf
# Original: $coordExtraConfig
include_if_exists = '/data/opentenbase/global/global_opentenbase.conf'
wal_level = replica
wal_keep_segments = 256
max_wal_senders = 4
archive_mode = on
archive_timeout = 1800
archive_command = 'echo 0'
log_truncate_on_rotation = on
log_filename = 'postgresql-%M.log'
log_rotation_age = 4h
log_rotation_size = 100MB
hot_standby = on
wal_sender_timeout = 30min
wal_receiver_timeout = 30min
shared_buffers = 1024MB
max_pool_size = 2000
log_statement = 'ddl'
log_destination = 'csvlog'
logging_collector = on
log_directory = 'pg_log'
listen_addresses = '*'
max_connections = 2000
EOF

coordSpecificExtraConfig=(none none)
coordExtraPgHba=coordExtraPgHba
cat > $coordExtraPgHba <<EOF
local all all trust
host all all 0.0.0.0/0 trust
host replication all 0.0.0.0/0 trust
host all all ::1/128 trust
host replication all ::1/128 trust
EOF

coordSpecificExtraPgHba=(none none)
coordAdditionalSlaves=n
cad1_Sync=n

# change: 3DN, so add the 3rd configuration

#---- Datanodes ---------------------
dn1MstrDir=/data/opentenbase/data/dn001
dn2MstrDir=/data/opentenbase/data/dn002
dn3MstrDir=/data/opentenbase/data/dn003
dn1SlvDir=/data/opentenbase/data/dn001
dn2SlvDir=/data/opentenbase/data/dn002
dn3SlvDir=/data/opentenbase/data/dn003
dn1ALDir=/data/opentenbase/data/datanode_archlog
dn2ALDir=/data/opentenbase/data/datanode_archlog
dn3ALDir=/data/opentenbase/data/datanode_archlog

primaryDatanode=dn001
datanodeNames=(dn001 dn002 dn003)
datanodePorts=(40004 40004 40004)
datanodePoolerPorts=(41110 41110 41110)
datanodePgHbaEntries=(0.0.0.0/0)
datanodeMasterServers=($IP_1 $IP_2 $IP_3)
datanodeMasterDirs=($dn1MstrDir $dn2MstrDir $dn3MstrDir)
dnWALSndr=4
datanodeMaxWALSenders=($dnWALSndr $dnWALSndr $dnWALSndr)

datanodeSlave=y
datanodeSlaveServers=($IP_2 $IP_3 $IP_1)
datanodeSlavePorts=(50004 54004 58004)
datanodeSlavePoolerPorts=(51110 51110 51110)
datanodeSlaveSync=n
datanodeSlaveDirs=($dn1SlvDir $dn2SlvDir $dn3SlvDir)
datanodeArchLogDirs=($dn1ALDir/dn001 $dn2ALDir/dn002 $dn3ALDir/dn003)

datanodeExtraConfig=datanodeExtraConfig
cat > $datanodeExtraConfig <<EOF
#================================================
# Added to all the coordinator postgresql.conf
# Original: $datanodeExtraConfig
include_if_exists = '/data/opentenbase/global/global_opentenbase.conf'
listen_addresses = '*'
wal_level = replica
wal_keep_segments = 256
max_wal_senders = 4
archive_mode = on
archive_timeout = 1800
archive_command = 'echo 0'
log_directory = 'pg_log'
logging_collector = on
log_truncate_on_rotation = on
log_filename = 'postgresql-%M.log'
log_rotation_age = 4h
log_rotation_size = 100MB
hot_standby = on
wal_sender_timeout = 30min
wal_receiver_timeout = 30min
shared_buffers = 1024MB
max_connections = 4000
max_pool_size = 4000
log_statement = 'ddl'
log_destination = 'csvlog'
wal_buffers = 1GB
EOF

# change: 3DN, so add a none
datanodeSpecificExtraConfig=(none none none)
datanodeExtraPgHba=datanodeExtraPgHba
cat > $datanodeExtraPgHba <<EOF
local all all trust
host all all 0.0.0.0/0 trust
host replication all 0.0.0.0/0 trust
host all all ::1/128 trust
host replication all ::1/128 trust
EOF

# change: 3DN, so add a none
datanodeSpecificExtraPgHba=(none none none)

datanodeAdditionalSlaves=n
walArchive=n

3 分发二进制包

在一个节点配置好配置文件后，需要预先将二进制包部署到所有节点所在的机器上，这个可以使用pgxc_ctl工具，执行deploy all命令来完成。

如果找不到命令：

Bash
# 查找 pgxc_ctl 的安装路径（通常在安装目录的 bin 下）
find / -name "pgxc_ctl" 2>/dev/null

# 找到路径，将其加入环境变量
echo 'export PATH=$PATH:/data/opentenbase/install/opentenbase_bin_v2.0/bin/pgxc_ctl' >> ~/.bashrc
source ~/.bashrc

部署成功：