Gbase8a测试技术分享--fio工具测试（磁盘性能）
一、fio工具安装
1. 获取安装包，解压并进行编译
#解压
tar -xvf fio-fio-3.25.tar
#编译
./configure
#安装
make
make install
二、fio参数配置
1.fio测试
#运行命令：
fio fio_1m.conf --output ${result_path}/fio_1m_result
注：
#--output 是将fio的运行结果统计到1个文件中，方便之后的统计与查看fio测试结果
#由于fio测试的测试点和参数配置比较多，可以将参数和各个测试场景写成如下的配置文件：
[global]
ioengine=libaio
iodepth=8
time_based
direct=1
group_reporting
randrepeat=0
norandommap
numjobs=16
timeout=6000
runtime=120
stonewall
filename=/opt/fiodata4
size=4G
[randread-1M]
rw=randread
bs=1M
rwmixread=100
[randwrite-1M]
rw=randwrite
bs=1M
[randrw-1M]
rw=randrw
bs=1M
rwmixread=50
[read-1M]
rw=read
bs=1M
rwmixread=100
[write-1M]
rw=write
bs=1M
[rw-1M]
rw=rw
bs=1M
rwmixread=50
其中，个人觉得重要关注参数：
（1）ioengine=libaio，在使用fio进行测试之前，进行libaio的安装，libaio是异步io模式，采用libaio更有效率；
（2）测试时的IO队列深度：iodepth=8
（3）测试运行时间：runtime=120
（4）测试io操作数量：size=4G
（5）io块大小：bs=1M
（6）主要的测试点：rw参数设置以下测试点（测试读写策略）
a.随机读：rw=randread，rwmixread=100，可以依据测试需求设置相关比例
b.随机写：rw=randwrite
c.随机读写：rw=randrw
d.顺序读：rw=read
e.顺序写：rw=write
f.顺序读写：rw=rw
三、测试结果
1.查看结果文件：
randread-1M: (g=0): rw=randread, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=8
...
randwrite-1M: (g=1): rw=randwrite, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=8
...
randrw-1M: (g=2): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=8
...
read-1M: (g=3): rw=read, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=8
...
write-1M: (g=4): rw=write, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=8
...
rw-1M: (g=5): rw=rw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=8
...
fio-3.25
Starting 96 processes
randread-1M: Laying out IO file (1 file / 4096MiB)
randread-1M: (groupid=0, jobs=16): err= 0: pid=60145: Mon Jul 4 14:45:43 2022
read: IOPS=6675, BW=6675MiB/s (7000MB/s)(783GiB/120048msec)
slat (usec): min=3, max=365, avg= 6.32, stdev= 1.95
clat (usec): min=177, max=1877.1k, avg=19162.56, stdev=27589.78
lat (usec): min=184, max=1877.1k, avg=19169.00, stdev=27589.84
clat percentiles (usec):
| 1.00th=[ 326], 5.00th=[ 330], 10.00th=[ 338], 20.00th=[ 619],
| 30.00th=[ 8356], 40.00th=[ 11863], 50.00th=[ 14877], 60.00th=[ 18220],
| 70.00th=[ 22676], 80.00th=[ 28443], 90.00th=[ 39060], 95.00th=[ 51119],
| 99.00th=[105382], 99.50th=[149947], 99.90th=[333448], 99.95th=[467665],
| 99.99th=[750781]
bw ( MiB/s): min= 1983, max= 8454, per=100.00%, avg=6679.31, stdev=54.41, samples=3827
iops : min= 1982, max= 8454, avg=6677.47, stdev=54.41, samples=3827
lat (usec) : 250=0.09%, 500=15.85%, 750=6.18%, 1000=1.74%
lat (msec) : 2=1.13%, 4=0.50%, 10=8.72%, 20=30.15%, 50=30.40%
lat (msec) : 100=4.13%, 250=0.91%, 500=0.15%, 750=0.03%, 1000=0.01%
lat (msec) : 2000=0.01%
cpu : usr=0.08%, sys=0.26%, ctx=797273, majf=0, minf=2184
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=801373,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=8
randwrite-1M: (groupid=1, jobs=16): err= 0: pid=60219: Mon Jul 4 14:45:43 2022
write: IOPS=1047, BW=1048MiB/s (1098MB/s)(123GiB/120439msec); 0 zone resets
slat (usec): min=20, max=446, avg=36.55, stdev= 9.13
clat (usec): min=617, max=1585.8k, avg=122016.37, stdev=142263.19
lat (usec): min=722, max=1585.8k, avg=122053.14, stdev=142263.08
clat percentiles (msec):
| 1.00th=[ 27], 5.00th=[ 43], 10.00th=[ 50], 20.00th=[ 58],
| 30.00th=[ 65], 40.00th=[ 70], 50.00th=[ 75], 60.00th=[ 82],
| 70.00th=[ 90], 80.00th=[ 107], 90.00th=[ 279], 95.00th=[ 451],
| 99.00th=[ 743], 99.50th=[ 844], 99.90th=[ 1053], 99.95th=[ 1150],
| 99.99th=[ 1401]
bw ( MiB/s): min= 466, max= 1934, per=100.00%, avg=1050.57, stdev=13.55, samples=3840
iops : min= 466, max= 1928, avg=1049.83, stdev=13.53, samples=3840
lat (usec) : 750=0.01%
lat (msec) : 2=0.01%, 4=0.02%, 10=0.04%, 20=0.31%, 50=10.13%
lat (msec) : 100=66.77%, 250=11.71%, 500=6.99%, 750=3.08%, 1000=0.79%
lat (msec) : 2000=0.16%
cpu : usr=0.20%, sys=0.07%, ctx=125805, majf=0, minf=134
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=99.9%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=0,126160,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=8
randrw-1M: (groupid=2, jobs=16): err= 0: pid=60280: Mon Jul 4 14:45:43 2022
read: IOPS=723, BW=724MiB/s (759MB/s)(84.0GiB/120262msec)
slat (nsec): min=5280, max=84880, avg=7594.06, stdev=1927.30
clat (usec): min=177, max=2394.3k, avg=124903.88, stdev=168975.09
lat (usec): min=185, max=2394.4k, avg=124911.63, stdev=168975.10
clat percentiles (usec):
| 1.00th=[ 186], 5.00th=[ 206], 10.00th=[ 445],
| 20.00th=[ 5735], 30.00th=[ 11994], 40.00th=[ 24249],
| 50.00th=[ 54264], 60.00th=[ 100140], 70.00th=[ 154141],
| 80.00th=[ 223347], 90.00th=[ 337642], 95.00th=[ 467665],
| 99.00th=[ 767558], 99.50th=[ 884999], 99.90th=[1182794],
| 99.95th=[1317012], 99.99th=[1652556]
bw ( KiB/s): min=233548, max=1409564, per=100.00%, avg=742121.64, stdev=13386.49, samples=3828
iops : min= 228, max= 1376, avg=724.27, stdev=13.07, samples=3828
write: IOPS=720, BW=720MiB/s (755MB/s)(84.6GiB/120262msec); 0 zone resets
slat (usec): min=16, max=438, avg=29.71, stdev= 6.39
clat (usec): min=188, max=2351.1k, avg=52039.49, stdev=154518.48
lat (usec): min=205, max=2351.2k, avg=52069.38, stdev=154518.47
clat percentiles (usec):
| 1.00th=[ 194], 5.00th=[ 198], 10.00th=[ 200],
| 20.00th=[ 206], 30.00th=[ 210], 40.00th=[ 212],
| 50.00th=[ 221], 60.00th=[ 289], 70.00th=[ 396],
| 80.00th=[ 955], 90.00th=[ 189793], 95.00th=[ 396362],
| 99.00th=[ 767558], 99.50th=[ 893387], 99.90th=[1199571],
| 99.95th=[1350566], 99.99th=[1820328]
bw ( KiB/s): min=157772, max=1608495, per=100.00%, avg=739231.79, stdev=16305.79, samples=3828
iops : min= 154, max= 1570, avg=721.46, stdev=15.91, samples=3828
lat (usec) : 250=31.53%, 500=10.05%, 750=2.10%, 1000=2.24%
lat (msec) : 2=1.39%, 4=2.88%, 10=5.10%, 20=5.32%, 50=6.30%
lat (msec) : 100=6.48%, 250=14.06%, 500=8.81%, 750=2.67%, 1000=0.81%
lat (msec) : 2000=0.27%, >=2000=0.01%
cpu : usr=0.12%, sys=0.08%, ctx=173440, majf=0, minf=140
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=99.9%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=87032,86625,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=8
read-1M: (groupid=3, jobs=16): err= 0: pid=60375: Mon Jul 4 14:45:43 2022
read: IOPS=6806, BW=6807MiB/s (7138MB/s)(798GiB/120024msec)
slat (usec): min=3, max=393, avg= 5.95, stdev= 2.21
clat (usec): min=178, max=885440, avg=18795.75, stdev=14582.01
lat (usec): min=184, max=885446, avg=18801.81, stdev=14582.03
clat percentiles (usec):
| 1.00th=[ 1319], 5.00th=[ 4359], 10.00th=[ 5997], 20.00th=[ 8029],
| 30.00th=[ 9896], 40.00th=[ 12649], 50.00th=[ 16909], 60.00th=[ 21365],
| 70.00th=[ 25297], 80.00th=[ 28967], 90.00th=[ 33162], 95.00th=[ 36439],
| 99.00th=[ 47973], 99.50th=[ 60031], 99.90th=[127402], 99.95th=[223347],
| 99.99th=[488637]
bw ( MiB/s): min= 3513, max= 9023, per=100.00%, avg=6808.80, stdev=40.18, samples=3840
iops : min= 3510, max= 9023, avg=6806.93, stdev=40.18, samples=3840
lat (usec) : 250=0.03%, 500=0.37%, 750=0.28%, 1000=0.15%
lat (msec) : 2=0.47%, 4=2.89%, 10=26.05%, 20=26.30%, 50=42.60%
lat (msec) : 100=0.71%, 250=0.11%, 500=0.03%, 750=0.01%, 1000=0.01%
cpu : usr=0.10%, sys=0.23%, ctx=813496, majf=0, minf=2199
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=816998,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=8
write-1M: (groupid=4, jobs=16): err= 0: pid=60466: Mon Jul 4 14:45:43 2022
write: IOPS=5335, BW=5335MiB/s (5594MB/s)(625GiB/120016msec); 0 zone resets
slat (usec): min=16, max=445, avg=33.32, stdev= 6.39
clat (usec): min=191, max=356636, avg=23955.63, stdev=19649.16
lat (usec): min=210, max=356669, avg=23989.17, stdev=19649.17
clat percentiles (usec):
| 1.00th=[ 338], 5.00th=[ 388], 10.00th=[ 523], 20.00th=[ 848],
| 30.00th=[ 25560], 40.00th=[ 28443], 50.00th=[ 28967], 60.00th=[ 29492],
| 70.00th=[ 30540], 80.00th=[ 31327], 90.00th=[ 32113], 95.00th=[ 36963],
| 99.00th=[113771], 99.50th=[123208], 99.90th=[193987], 99.95th=[227541],
| 99.99th=[283116]
bw ( MiB/s): min= 3116, max= 6606, per=100.00%, avg=5343.87, stdev=49.99, samples=3824
iops : min= 3116, max= 6602, avg=5340.03, stdev=49.97, samples=3824
lat (usec) : 250=0.49%, 500=8.81%, 750=8.52%, 1000=4.56%
lat (msec) : 2=3.96%, 4=0.41%, 10=0.57%, 20=0.98%, 50=69.22%
lat (msec) : 100=1.17%, 250=1.29%, 500=0.03%
cpu : usr=0.90%, sys=0.37%, ctx=639949, majf=0, minf=184
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=0,640298,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=8
rw-1M: (groupid=5, jobs=16): err= 0: pid=60495: Mon Jul 4 14:45:43 2022
read: IOPS=2487, BW=2488MiB/s (2609MB/s)(292GiB/120226msec)
slat (usec): min=4, max=116, avg= 8.00, stdev= 2.62
clat (usec): min=176, max=1211.6k, avg=33911.27, stdev=54771.63
lat (usec): min=182, max=1211.6k, avg=33919.42, stdev=54771.64
clat percentiles (usec):
| 1.00th=[ 227], 5.00th=[ 355], 10.00th=[ 502], 20.00th=[ 1156],
| 30.00th=[ 4490], 40.00th=[ 10159], 50.00th=[ 16712], 60.00th=[ 21365],
| 70.00th=[ 27395], 80.00th=[ 47449], 90.00th=[ 94897], 95.00th=[143655],
| 99.00th=[263193], 99.50th=[312476], 99.90th=[446694], 99.95th=[509608],
| 99.99th=[734004]
bw ( MiB/s): min= 578, max= 5750, per=100.00%, avg=2492.07, stdev=58.59, samples=3838
iops : min= 578, max= 5750, avg=2492.04, stdev=58.59, samples=3838
write: IOPS=2469, BW=2470MiB/s (2590MB/s)(290GiB/120226msec); 0 zone resets
slat (usec): min=16, max=162, avg=31.75, stdev= 5.62
clat (usec): min=189, max=1070.9k, avg=17576.80, stdev=39486.63
lat (usec): min=206, max=1070.9k, avg=17608.75, stdev=39486.99
clat percentiles (usec):
| 1.00th=[ 200], 5.00th=[ 219], 10.00th=[ 289], 20.00th=[ 457],
| 30.00th=[ 947], 40.00th=[ 2900], 50.00th=[ 5932], 60.00th=[ 11076],
| 70.00th=[ 17433], 80.00th=[ 21890], 90.00th=[ 31327], 95.00th=[ 68682],
| 99.00th=[210764], 99.50th=[267387], 99.90th=[404751], 99.95th=[471860],
| 99.99th=[658506]
bw ( MiB/s): min= 488, max= 5794, per=100.00%, avg=2474.31, stdev=59.73, samples=3838
iops : min= 488, max= 5794, avg=2474.28, stdev=59.74, samples=3838
lat (usec) : 250=4.67%, 500=11.02%, 750=5.70%, 1000=3.24%
lat (msec) : 2=5.67%, 4=6.02%, 10=12.75%, 20=16.52%, 50=21.60%
lat (msec) : 100=6.39%, 250=5.53%, 500=0.86%, 750=0.04%, 1000=0.01%
lat (msec) : 2000=0.01%
cpu : usr=0.44%, sys=0.28%, ctx=589087, majf=0, minf=156
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=299108,296926,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=8
Run status group 0 (all jobs):
READ: bw=6675MiB/s (7000MB/s), 6675MiB/s-6675MiB/s (7000MB/s-7000MB/s), io=783GiB (840GB), run=120048-120048msec
Run status group 1 (all jobs):
WRITE: bw=1048MiB/s (1098MB/s), 1048MiB/s-1048MiB/s (1098MB/s-1098MB/s), io=123GiB (132GB), run=120439-120439msec
Run status group 2 (all jobs):
READ: bw=724MiB/s (759MB/s), 724MiB/s-724MiB/s (759MB/s-759MB/s), io=84.0GiB (91.3GB), run=120262-120262msec
WRITE: bw=720MiB/s (755MB/s), 720MiB/s-720MiB/s (755MB/s-755MB/s), io=84.6GiB (90.8GB), run=120262-120262msec
Run status group 3 (all jobs):
READ: bw=6807MiB/s (7138MB/s), 6807MiB/s-6807MiB/s (7138MB/s-7138MB/s), io=798GiB (857GB), run=120024-120024msec
Run status group 4 (all jobs):
WRITE: bw=5335MiB/s (5594MB/s), 5335MiB/s-5335MiB/s (5594MB/s-5594MB/s), io=625GiB (671GB), run=120016-120016msec
Run status group 5 (all jobs):
READ: bw=2488MiB/s (2609MB/s), 2488MiB/s-2488MiB/s (2609MB/s-2609MB/s), io=292GiB (314GB), run=120226-120226msec
WRITE: bw=2470MiB/s (2590MB/s), 2470MiB/s-2470MiB/s (2590MB/s-2590MB/s), io=290GiB (311GB), run=120226-120226msec
Disk stats (read/write):
sdb: ios=2004486/1150072, merge=0/24, ticks=51710365/40427723, in_queue=92811880, util=100.00%
2. 结果关注的两个参数
以其中一个测试策略：随机读 为例：
a. Iops：每秒进行io操作次数
b. Bw：线程带宽
总结：
io结果文件需要关注统计的参数只有一部分，那么怎样获取到我们想要统计的结果？
一般测试点涵盖bs=1M，4k，64k，512k，4M，如何编写脚本进行fio测试？
