logstash报400错误

logstash的报错信息如下：

[2022-11-11T03:11:31,141][ERROR][logstash.outputs.elasticsearch][main][c2ad172f52843e77a0a839e66a0bed2f1fdb77cecbdc75ba37847c9284625c0d] Encountered a retryable error. Will Retry with exponential backoff  {:code=>400, :url=>"https://30.9.0.85:9200/_bulk", :body=>"{\"error\":{\"root_cause\":[{\"type\":\"parse_exception\",\"reason\":\"request body is required\"}],\"type\":\"parse_exception\",\"reason\":\"request body is required\"},\"status\":400}"}

官方文档说明

[Elasticsearch output plugin | Logstash Reference 7.9] | Elastic

HTTP Compression
This plugin supports request and response compression. Response compression is enabled by default for HTTP and for Elasticsearch versions 5.0 and later.

You don’t have to set any configs in Elasticsearch for it to send back a compressed response. For versions before 5.0, or if HTTPS is enabled, http.compression must be set to true in Elasticsearch to take advantage of response compression when using this plugin.

For requests compression, regardless of the Elasticsearch version, enable the http_compression setting in the Logstash config file.

custom_headers
Value type is hash
There is no default value for this setting.
Pass a set of key value pairs as the headers sent in each request to an elasticsearch node. The headers will be used for any kind of request (_bulk request, template installation, health checks and sniffing). These custom headers will be overidden by settings like http_compression.

AWS关于这个参数的说明

HTTP 压缩支持现已在 Amazon Elasticsearch Service 中提供

HTTP 压缩支持现已在 Amazon Elasticsearch Service 中提供
发布于: Jul 28, 2020

Amazon Elasticsearch Service 现在支持对 HTTP 负载进行 gzip 压缩。使用 HTTP 压缩，您最多可以将您的文档大小缩小 80%，从而降低带宽利用率和延迟，继而提高传输速度。  

我们推出了一个动态启用或禁用 HTTP 压缩的新集群设置，无需重启您的域。HTTP 压缩使用标准的 gzip 文件格式。要发送和接收压缩内容，客户端的请求和响应中必须有相关的 HTTP 标头。现有客户端将继续按原样工作。 

HTTP 压缩功能现在可用于运行 Elasticsearch 6.0 版及以上的所有域。要了解更多信息，请参阅文档。

Amazon Elasticsearch Service 的 HTTP 压缩功能现已在全球 24 个区域推出，包括：美国东部（弗吉尼亚北部、俄亥俄）、美国西部（俄勒冈、加利福尼亚北部）、AWS GovCloud（US-Gov-East、US-Gov-West）、加拿大（中部）、南美洲（圣保罗）、欧洲（爱尔兰、伦敦、法兰克福、巴黎、斯德哥尔摩、米兰）、亚太地区（新加坡、悉尼、东京、首尔、孟买、香港）、中东（巴林）、中国（北京 – 由光环新网运营、宁夏 – 由西云数据运营）、非洲（开普敦）。有关 Amazon Elasticsearch Service 可用性的更多信息，请参阅AWS 区域表。

按照AWS的介绍，我理解开启http_compression这个参数后，发送到Elasticsearch的http数据包必须包含请求体和请求头信息，所以将此参数注释掉，再继续观察看问题是否出现。

Logstash配置如下

input {
  tcp {
    host => "0.0.0.0" port => 9250 mode => "server" tags => ["tags"] codec => "json_lines" tcp_keep_alive => "false"
  }
}

filter {
   mutate{
      lowercase => ["appname"]
   }
   truncate{
      fields => "message"
      length_bytes => 80000000
   }
}

output {
  if "aeonlife-gateway" in [appname] and "aeonlife-gateway-wan" not in [appname] or "aeonlife-gatewayapi-service" in [appname] or "aeonlife-file"  in [appname] or "aeonlife-ocr"  in [appname] {
      elasticsearch {
          hosts => [ "https://30.9.1.168:9200","https://30.9.3.187:9200","https://30.9.0.85:9200","https://30.9.3.1:9200","https://30.9.2.207:9200","https://30.9.1.12:9200","https://30.9.3.248:9200","https://30.9.2.66:9200" ]
          index => "%{[appname]}-%{+YYYY.MM.dd}"
          retry_on_conflict => 2
          sniffing => "false"
          user => "logstash_tcpuser"
          password => "##########"
          ilm_enabled => false
          cacert => "/usr/local/logstash-7.9.3/config/CloudSearchService.cer"
#          http_compression => "true"
          ssl => true
      }
  }
  else {
      elasticsearch {
          hosts => [ "https://30.9.1.168:9200","https://30.9.3.187:9200","https://30.9.0.85:9200","https://30.9.3.1:9200","https://30.9.2.207:9200","https://30.9.1.12:9200","https://30.9.3.248:9200","https://30.9.2.66:9200" ]
          index => "logstash-%{+YYYY.MM.dd}"
          retry_on_conflict => 2
          sniffing => "false"
          user => "logstash_tcpuser"
          password => ""##########"
          ilm_enabled => false
          cacert => "/usr/local/logstash-7.9.3/config/CloudSearchService.cer"
#          http_compression => "true"
          ssl => true
      }
  }
}