解决Doris通过broker导入数据报错：Scan bytes per file scanner exceed limit: 5368709120

报错问题

报错显示:ErrorMsg: type:ETL_RUN_FAIL; msg:errCode = 2, detailMessage = Scan bytes per file scanner exceed limit: 5368709120
详细的报错信息显示如下：

mysql> show load order by createtime desc limit 1\G;
*************************** 1. row ***************************
         JobId: 21644
         Label: partsupp
         State: CANCELLED
      Progress: ETL:N/A; LOAD:N/A
          Type: BROKER
       EtlInfo: NULL
      TaskInfo: cluster:N/A; timeout(s):3600; max_filter_ratio:0.1
      ErrorMsg: type:ETL_RUN_FAIL; msg:errCode = 2, detailMessage = Scan bytes per file scanner exceed limit: 5368709120
    CreateTime: 2023-02-03 03:34:32
  EtlStartTime: NULL
 EtlFinishTime: NULL
 LoadStartTime: NULL
LoadFinishTime: 2023-02-03 03:34:34
           URL: NULL
    JobDetails: {"Unfinished backends":{},"ScannedRows":0,"TaskNumber":0,"LoadBytes":0,"All backends":{},"FileNumber":9,"FileSize":125871992078}
 TransactionId: 10030
  ErrorTablets: {}
1 row in set (0.01 sec)

执行的导入脚本

LOAD LABEL tpch1t.orders
(
  DATA INFILE("hdfs://192.168.30.71:9010/tpch1t/partsupp/*")
  INTO TABLE `orders`
  COLUMNS TERMINATED BY "|"            
  (o_orderkey,o_orderdate,o_custkey,o_orderstatus,o_totalprice,o_orderpriority,o_clerk,o_shippriority,o_comment)
)
WITH BROKER "broker_name" 
    ( 
      "username" = "root", 
      "password" = "" 
    )
PROPERTIES
(
    "timeout"="3600",
    "max_filter_ratio"="0.1"
);

解决方法

该Doris测试环境为3个BE节点，而FE的配置max_bytes_per_broker_scanner默认为3G，所要导入的文件共有大约120GB，需要修改参数FE执行如下动态修改参数命令：

ADMIN SET FRONTEND CONFIG ("max_bytes_per_broker_scanner" = "42949672960");

参数值修改为40G，这样一样来，集群最大可导入的文件大小为40G * 3(个BE)=120GB，再次执行即可。

单位换算方法

工具推荐链接：https://www.elecfans.com/tools/zijiehuansuan.html