Elasticsearch集群413错误

logstash日志如下：

[2024-07-04T08:31:51,093][ERROR][logstash.outputs.elasticsearch][main] Encountered a retryable error. Will Retry with exponential backoff  {:code=>413, :url=>"https://10.1.52.114:9200/_bulk"}

重启logstash日志如下：

[2024-07-04T09:02:31,561][ERROR][logstash.outputs.elasticsearch][main] Encountered a retryable error. Will Retry with exponential backoff  {:code=>413, :url=>"https://10.1.52.114:9200/_bulk"}
[2024-07-04T09:02:36,343][WARN ][org.logstash.execution.ShutdownWatcherExt] {"inflight_count"=>0, "stalling_threads_info"=>{"other"=>[{"thread_id"=>84, "name"=>"[main]<beats", "current_call"=>
"[...]/vendor/bundle/jruby/2.5.0/gems/logstash-input-beats-6.0.9-java/lib/logstash/inputs/beats.rb:197:in `run'"}], ["LogStash::Filters::Mutate", {"remove_field"=>["@version", "cloud", "ecs", 
"host"], "id"=>"13837525356b64316d83248c4838943a405fcea76ce219e9fed12f1719b7749d"}]=>[{"thread_id"=>81, "name"=>"[main]>worker0", "current_call"=>"[...]/vendor/bundle/jruby/2.5.0/gems/stud-0.0
.23/lib/stud/interval.rb:89:in `sleep'"}, {"thread_id"=>82, "name"=>"[main]>worker1", "current_call"=>"[...]/vendor/bundle/jruby/2.5.0/gems/stud-0.0.23/lib/stud/interval.rb:89:in `sleep'"}]}}
[2024-07-04T09:02:41,361][WARN ][org.logstash.execution.ShutdownWatcherExt] {"inflight_count"=>0, "stalling_threads_info"=>{"other"=>[{"thread_id"=>84, "name"=>"[main]<beats", "current_call"=>
"[...]/vendor/bundle/jruby/2.5.0/gems/logstash-input-beats-6.0.9-java/lib/logstash/inputs/beats.rb:197:in `run'"}], ["LogStash::Filters::Mutate", {"remove_field"=>["@version", "cloud", "ecs", 
"host"], "id"=>"13837525356b64316d83248c4838943a405fcea76ce219e9fed12f1719b7749d"}]=>[{"thread_id"=>81, "name"=>"[main]>worker0", "current_call"=>"[...]/vendor/bundle/jruby/2.5.0/gems/stud-0.0
.23/lib/stud/interval.rb:89:in `sleep'"}, {"thread_id"=>82, "name"=>"[main]>worker1", "current_call"=>"[...]/vendor/bundle/jruby/2.5.0/gems/stud-0.0.23/lib/stud/interval.rb:89:in `sleep'"}]}}
[2024-07-04T09:02:46,378][WARN ][org.logstash.execution.ShutdownWatcherExt] {"inflight_count"=>0, "stalling_threads_info"=>{"other"=>[{"thread_id"=>84, "name"=>"[main]<beats", "current_call"=>
"[...]/vendor/bundle/jruby/2.5.0/gems/logstash-input-beats-6.0.9-java/lib/logstash/inputs/beats.rb:197:in `run'"}], ["LogStash::Filters::Mutate", {"remove_field"=>["@version", "cloud", "ecs", 
"host"], "id"=>"13837525356b64316d83248c4838943a405fcea76ce219e9fed12f1719b7749d"}]=>[{"thread_id"=>81, "name"=>"[main]>worker0", "current_call"=>"[...]/vendor/bundle/jruby/2.5.0/gems/stud-0.0
.23/lib/stud/interval.rb:89:in `sleep'"}, {"thread_id"=>82, "name"=>"[main]>worker1", "current_call"=>"[...]/vendor/bundle/jruby/2.5.0/gems/stud-0.0.23/lib/stud/interval.rb:89:in `sleep'"}]}}
[2024-07-04T09:02:51,397][WARN ][org.logstash.execution.ShutdownWatcherExt] {"inflight_count"=>0, "stalling_threads_info"=>{"other"=>[{"thread_id"=>84, "name"=>"[main]<beats", "current_call"=>
"[...]/vendor/bundle/jruby/2.5.0/gems/logstash-input-beats-6.0.9-java/lib/logstash/inputs/beats.rb:197:in `run'"}], ["LogStash::Filters::Mutate", {"remove_field"=>["@version", "cloud", "ecs", 
"host"], "id"=>"13837525356b64316d83248c4838943a405fcea76ce219e9fed12f1719b7749d"}]=>[{"thread_id"=>81, "name"=>"[main]>worker0", "current_call"=>"[...]/vendor/bundle/jruby/2.5.0/gems/stud-0.0
.23/lib/stud/interval.rb:89:in `sleep'"}, {"thread_id"=>82, "name"=>"[main]>worker1", "current_call"=>"[...]/vendor/bundle/jruby/2.5.0/gems/stud-0.0.23/lib/stud/interval.rb:89:in `sleep'"}]}}
[2024-07-04T09:02:56,420][WARN ][org.logstash.execution.ShutdownWatcherExt] {"inflight_count"=>0, "stalling_threads_info"=>{"other"=>[{"thread_id"=>84, "name"=>"[main]<beats", "current_call"=>
"[...]/vendor/bundle/jruby/2.5.0/gems/logstash-input-beats-6.0.9-java/lib/logstash/inputs/beats.rb:197:in `run'"}], ["LogStash::Filters::Mutate", {"remove_field"=>["@version", "cloud", "ecs", 
"host"], "id"=>"13837525356b64316d83248c4838943a405fcea76ce219e9fed12f1719b7749d"}]=>[{"thread_id"=>81, "name"=>"[main]>worker0", "current_call"=>"[...]/vendor/bundle/jruby/2.5.0/gems/manticor
e-0.6.4-java/lib/manticore/response.rb:50:in `call'"}, {"thread_id"=>82, "name"=>"[main]>worker1", "current_call"=>"[...]/vendor/bundle/jruby/2.5.0/gems/stud-0.0.23/lib/stud/interval.rb:89:in 
`sleep'"}]}}
[2024-07-04T09:02:57,146][ERROR][logstash.outputs.elasticsearch][main] Encountered a retryable error. Will Retry with exponential backoff  {:code=>413, :url=>"https://10.1.53.197:9200/_bulk"}
[2024-07-04T09:03:01,439][WARN ][org.logstash.execution.ShutdownWatcherExt] {"inflight_count"=>0, "stalling_threads_info"=>{"other"=>[{"thread_id"=>84, "name"=>"[main]<beats", "current_call"=>
"[...]/vendor/bundle/jruby/2.5.0/gems/logstash-input-beats-6.0.9-java/lib/logstash/inputs/beats.rb:197:in `run'"}], ["LogStash::Filters::Mutate", {"remove_field"=>["@version", "cloud", "ecs", 
"host"], "id"=>"13837525356b64316d83248c4838943a405fcea76ce219e9fed12f1719b7749d"}]=>[{"thread_id"=>81, "name"=>"[main]>worker0", "current_call"=>"[...]/vendor/bundle/jruby/2.5.0/gems/stud-0.0
.23/lib/stud/interval.rb:89:in `sleep'"}, {"thread_id"=>82, "name"=>"[main]>worker1", "current_call"=>"[...]/vendor/bundle/jruby/2.5.0/gems/stud-0.0.23/lib/stud/interval.rb:89:in `sleep'"}]}}
[2024-07-04T09:03:06,456][WARN ][org.logstash.execution.ShutdownWatcherExt] {"inflight_count"=>0, "stalling_threads_info"=>{"other"=>[{"thread_id"=>84, "name"=>"[main]<beats", "current_call"=>
"[...]/vendor/bundle/jruby/2.5.0/gems/logstash-input-beats-6.0.9-java/lib/logstash/inputs/beats.rb:197:in `run'"}], ["LogStash::Filters::Mutate", {"remove_field"=>["@version", "cloud", "ecs", 
"host"], "id"=>"13837525356b64316d83248c4838943a405fcea76ce219e9fed12f1719b7749d"}]=>[{"thread_id"=>81, "name"=>"[main]>worker0", "current_call"=>"[...]/vendor/bundle/jruby/2.5.0/gems/stud-0.0
.23/lib/stud/interval.rb:89:in `sleep'"}, {"thread_id"=>82, "name"=>"[main]>worker1", "current_call"=>"[...]/vendor/bundle/jruby/2.5.0/gems/stud-0.0.23/lib/stud/interval.rb:89:in `sleep'"}]}}
[2024-07-04T09:03:11,473][WARN ][org.logstash.execution.ShutdownWatcherExt] {"inflight_count"=>0, "stalling_threads_info"=>{"other"=>[{"thread_id"=>84, "name"=>"[main]<beats", "current_call"=>
"[...]/vendor/bundle/jruby/2.5.0/gems/logstash-input-beats-6.0.9-java/lib/logstash/inputs/beats.rb:197:in `run'"}], ["LogStash::Filters::Mutate", {"remove_field"=>["@version", "cloud", "ecs", 
"host"], "id"=>"13837525356b64316d83248c4838943a405fcea76ce219e9fed12f1719b7749d"}]=>[{"thread_id"=>81, "name"=>"[main]>worker0", "current_call"=>"[...]/vendor/bundle/jruby/2.5.0/gems/stud-0.0
.23/lib/stud/interval.rb:89:in `sleep'"}, {"thread_id"=>82, "name"=>"[main]>worker1", "current_call"=>"[...]/vendor/bundle/jruby/2.5.0/gems/stud-0.0.23/lib/stud/interval.rb:89:in `sleep'"}]}}
[2024-07-04T09:03:16,491][WARN ][org.logstash.execution.ShutdownWatcherExt] {"inflight_count"=>0, "stalling_threads_info"=>{"other"=>[{"thread_id"=>84, "name"=>"[main]<beats", "current_call"=>
"[...]/vendor/bundle/jruby/2.5.0/gems/logstash-input-beats-6.0.9-java/lib/logstash/inputs/beats.rb:197:in `run'"}], ["LogStash::Filters::Mutate", {"remove_field"=>["@version", "cloud", "ecs", 
"host"], "id"=>"13837525356b64316d83248c4838943a405fcea76ce219e9fed12f1719b7749d"}]=>[{"thread_id"=>81, "name"=>"[main]>worker0", "current_call"=>"[...]/vendor/bundle/jruby/2.5.0/gems/stud-0.0
.23/lib/stud/interval.rb:89:in `sleep'"}, {"thread_id"=>82, "name"=>"[main]>worker1", "current_call"=>"[...]/vendor/bundle/jruby/2.5.0/gems/stud-0.0.23/lib/stud/interval.rb:89:in `sleep'"}]}}
[2024-07-04T09:03:21,508][WARN ][org.logstash.execution.ShutdownWatcherExt] {"inflight_count"=>0, "stalling_threads_info"=>{"other"=>[{"thread_id"=>84, "name"=>"[main]<beats", "current_call"=>
"[...]/vendor/bundle/jruby/2.5.0/gems/logstash-input-beats-6.0.9-java/lib/logstash/inputs/beats.rb:197:in `run'"}], ["LogStash::Filters::Mutate", {"remove_field"=>["@version", "cloud", "ecs", 
"host"], "id"=>"13837525356b64316d83248c4838943a405fcea76ce219e9fed12f1719b7749d"}]=>[{"thread_id"=>81, "name"=>"[main]>worker0", "current_call"=>"[...]/vendor/bundle/jruby/2.5.0/gems/stud-0.0
.23/lib/stud/interval.rb:89:in `sleep'"}, {"thread_id"=>82, "name"=>"[main]>worker1", "current_call"=>"[...]/vendor/bundle/jruby/2.5.0/gems/stud-0.0.23/lib/stud/interval.rb:89:in `sleep'"}]}}
[2024-07-04T09:03:26,525][WARN ][org.logstash.execution.ShutdownWatcherExt] {"inflight_count"=>0, "stalling_threads_info"=>{"other"=>[{"thread_id"=>84, "name"=>"[main]<beats", "current_call"=>
"[...]/vendor/bundle/jruby/2.5.0/gems/logstash-input-beats-6.0.9-java/lib/logstash/inputs/beats.rb:197:in `run'"}], ["LogStash::Filters::Mutate", {"remove_field"=>["@version", "cloud", "ecs", 
"host"], "id"=>"13837525356b64316d83248c4838943a405fcea76ce219e9fed12f1719b7749d"}]=>[{"thread_id"=>81, "name"=>"[main]>worker0", "current_call"=>"[...]/vendor/bundle/jruby/2.5.0/gems/stud-0.0
.23/lib/stud/interval.rb:89:in `sleep'"}, {"thread_id"=>82, "name"=>"[main]>worker1", "current_call"=>"[...]/vendor/bundle/jruby/2.5.0/gems/stud-0.0.23/lib/stud/interval.rb:89:in `sleep'"}]}}
[2024-07-04T09:03:31,542][WARN ][org.logstash.execution.ShutdownWatcherExt] {"inflight_count"=>0, "stalling_threads_info"=>{"other"=>[{"thread_id"=>84, "name"=>"[main]<beats", "current_call"=>
"[...]/vendor/bundle/jruby/2.5.0/gems/logstash-input-beats-6.0.9-java/lib/logstash/inputs/beats.rb:197:in `run'"}], ["LogStash::Filters::Mutate", {"remove_field"=>["@version", "cloud", "ecs", 
"host"], "id"=>"13837525356b64316d83248c4838943a405fcea76ce219e9fed12f1719b7749d"}]=>[{"thread_id"=>81, "name"=>"[main]>worker0", "current_call"=>"[...]/vendor/bundle/jruby/2.5.0/gems/stud-0.0
.23/lib/stud/interval.rb:89:in `sleep'"}, {"thread_id"=>82, "name"=>"[main]>worker1", "current_call"=>"[...]/vendor/bundle/jruby/2.5.0/gems/stud-0.0.23/lib/stud/interval.rb:89:in `sleep'"}]}}
[2024-07-04T09:03:36,579][WARN ][org.logstash.execution.ShutdownWatcherExt] {"inflight_count"=>0, "stalling_threads_info"=>{"other"=>[{"thread_id"=>84, "name"=>"[main]<beats", "current_call"=>
"[...]/vendor/bundle/jruby/2.5.0/gems/logstash-input-beats-6.0.9-java/lib/logstash/inputs/beats.rb:197:in `run'"}], ["LogStash::Filters::Mutate", {"remove_field"=>["@version", "cloud", "ecs", 
"host"], "id"=>"13837525356b64316d83248c4838943a405fcea76ce219e9fed12f1719b7749d"}]=>[{"thread_id"=>81, "name"=>"[main]>worker0", "current_call"=>"[...]/vendor/bundle/jruby/2.5.0/gems/stud-0.0
.23/lib/stud/interval.rb:89:in `sleep'"}, {"thread_id"=>82, "name"=>"[main]>worker1", "current_call"=>"[...]/vendor/bundle/jruby/2.5.0/gems/manticore-0.6.4-java/lib/manticore/response.rb:50:in
 `call'"}]}}
[2024-07-04T09:03:38,213][ERROR][logstash.outputs.elasticsearch][main] Encountered a retryable error. Will Retry with exponential backoff  {:code=>413, :url=>"https://10.1.53.197:9200/_bulk"}
[2024-07-04T09:03:41,609][WARN ][org.logstash.execution.ShutdownWatcherExt] {"inflight_count"=>0, "stalling_threads_info"=>{"other"=>[{"thread_id"=>84, "name"=>"[main]<beats", "current_call"=>
"[...]/vendor/bundle/jruby/2.5.0/gems/logstash-input-beats-6.0.9-java/lib/logstash/inputs/beats.rb:197:in `run'"}], ["LogStash::Filters::Mutate", {"remove_field"=>["@version", "cloud", "ecs", 
"host"], "id"=>"13837525356b64316d83248c4838943a405fcea76ce219e9fed12f1719b7749d"}]=>[{"thread_id"=>81, "name"=>"[main]>worker0", "current_call"=>"[...]/vendor/bundle/jruby/2.5.0/gems/stud-0.0
.23/lib/stud/interval.rb:89:in `sleep'"}, {"thread_id"=>82, "name"=>"[main]>worker1", "current_call"=>"[...]/vendor/bundle/jruby/2.5.0/gems/stud-0.0.23/lib/stud/interval.rb:89:in `sleep'"}]}}
[2024-07-04T09:04:02,181][INFO ][logstash.runner          ] Starting Logstash {"logstash.version"=>"7.6.2"}
[2024-07-04T09:04:09,520][INFO ][org.reflections.Reflections] Reflections took 38 ms to scan 1 urls, producing 20 keys and 40 values 

解决方案：

配置集群参数然后重启elasticsearch集群，再重启logstash。
http.max_content_length: 500mb