glusterfs挂载与nfs挂载的性能差异

glusterfs的测试结果

挂载命令:
mount -t glusterfs xx.xx.xx68:/glusterdomain  gluster/test
测试命令:
fio -name=1io -filename=io.speedtest -runtime=30 -direct=1 -bs=8k -rw=write  -size=4G  -thread  -group_reporting  -numjobs=60  --iodepth=4 -ioengine=sync
测试结果:
fio-3.14
Starting 60 threads
Jobs: 60 (f=60): [W(60)][100.0%][w=52.8MiB/s][w=6754 IOPS][eta 00m:00s]
1io: (groupid=0, jobs=60): err= 0: pid=1444897: Tue Jul 30 08:19:30 2024
  write: IOPS=4590, BW=35.9MiB/s (37.6MB/s)(1077MiB/30022msec); 0 zone resets
    clat (usec): min=70, max=201645, avg=13049.31, stdev=16028.10
     lat (usec): min=71, max=201646, avg=13050.27, stdev=16028.13
    clat percentiles (usec):
     |  1.00th=[    79],  5.00th=[    84], 10.00th=[    86], 20.00th=[    91],
     | 30.00th=[    97], 40.00th=[   111], 50.00th=[   355], 60.00th=[ 22152],
     | 70.00th=[ 23725], 80.00th=[ 25822], 90.00th=[ 29754], 95.00th=[ 34866],
     | 99.00th=[ 61080], 99.50th=[ 79168], 99.90th=[114820], 99.95th=[115868],
     | 99.99th=[198181]
   bw (  KiB/s): min= 5644, max=196043, per=97.79%, avg=35909.43, stdev=1184.10, samples=3597
   iops        : min=  702, max=24505, avg=4486.23, stdev=148.02, samples=3597
  lat (usec)   : 100=33.38%, 250=13.69%, 500=6.70%, 750=0.18%, 1000=0.01%
  lat (msec)   : 2=0.01%, 4=0.02%, 10=0.03%, 20=0.56%, 50=43.88%
  lat (msec)   : 100=1.25%, 250=0.31%
  cpu          : usr=0.02%, sys=0.23%, ctx=413546, majf=0, minf=0
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,137812,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=4

Run status group 0 (all jobs):
  WRITE: bw=35.9MiB/s (37.6MB/s), 35.9MiB/s-35.9MiB/s (37.6MB/s-37.6MB/s), io=1077MiB (1129MB), run=30022-30022msec

nfs方式挂载

gluster volume info glusterdomain
gluster volume set glusterdomain nfs.disable off
需要注意:
Gluster NFS is being deprecated in favor of NFS-Ganesha Enter "yes" to continue using Gluster NFS (y/n)
NFS后续就不在支持了.
gluster volume stop glusterdomain && gluster volume start glusterdomain

# 注意建议所有节点都执行相关操作

mkdir -p gluster/nfs 
mount -t nfs node01:glusterdomain gluster/nfs

nfs测试结果

fio-3.14
Starting 60 threads
Jobs: 60 (f=60): [W(60)][100.0%][w=23.0MiB/s][w=2949 IOPS][eta 00m:00s]
1io: (groupid=0, jobs=60): err= 0: pid=1445801: Tue Jul 30 08:26:11 2024
  write: IOPS=3043, BW=23.8MiB/s (24.9MB/s)(714MiB/30019msec); 0 zone resets
    clat (usec): min=1534, max=261956, avg=19543.02, stdev=6349.79
     lat (usec): min=1534, max=261957, avg=19543.62, stdev=6349.80
    clat percentiles (msec):
     |  1.00th=[   15],  5.00th=[   19], 10.00th=[   19], 20.00th=[   19],
     | 30.00th=[   20], 40.00th=[   20], 50.00th=[   20], 60.00th=[   20],
     | 70.00th=[   20], 80.00th=[   21], 90.00th=[   21], 95.00th=[   21],
     | 99.00th=[   22], 99.50th=[   23], 99.90th=[   59], 99.95th=[  259],
     | 99.99th=[  262]
   bw (  KiB/s): min=11692, max=30608, per=100.00%, avg=24410.14, stdev=40.65, samples=3589
   iops        : min= 1458, max= 3826, avg=3048.43, stdev= 5.09, samples=3589
  lat (msec)   : 2=0.03%, 4=0.04%, 10=0.38%, 20=79.13%, 50=20.14%
  lat (msec)   : 100=0.20%, 250=0.01%, 500=0.06%
  cpu          : usr=0.02%, sys=0.11%, ctx=91809, majf=0, minf=0
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,91376,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=4

Run status group 0 (all jobs):
  WRITE: bw=23.8MiB/s (24.9MB/s), 23.8MiB/s-23.8MiB/s (24.9MB/s-24.9MB/s), io=714MiB (749MB), run=30019-30019msec

测试结论

1. NFS挂载只有 glustefs挂载方式性能的三分之二
   还是建议使用glusterfs的方式进行挂载
   建议还是使用rpm包的方式进行安装. 二进制编译没找到对应的挂载点. 

2. glusterfs 对nfs的支持马上就要废弃了, 建议还是使用glusterfs

3. glusterfs 虽然挂载之挂载一个节点, 但是理论上可以实现单节点故障时的高可用迁移.
   所以使用native原生驱动的效果更好一些.