python 3.7.6版

pyspark==2.4.4

在群集模式下运行，所有示例都已启动。
会话不是从与计算机相同的主机启动的。
已经重新启动了所有主进程和工作进程，因此问题必须在api机器中。
通过flaskapi运行它，创建一个会话，并为所有作业一直保持它的活动状态。
显然，应用程序在低工作负载下运行良好，但当增加请求数时，它会陷入以下错误：
在处理上述异常时，发生了另一个异常：

Traceback (most recent call last):
  File "/source/environment/lib/python3.7/site-packages/py4j/java_gateway.py", line 1067, in start
    self.socket.connect((self.address, self.port))
ConnectionRefusedError: [Errno 111] Connection refused
[pid: 2853|app: 0|req: 10/16] 172.31.59.3 () {32 vars in 457 bytes} [Mon Jul 13 09:24:16 2020] POST 
ERROR:py4j.java_gateway:An error occurred while trying to connect to the Java server (127.0.0.1:40277)
Traceback (most recent call last):
  File "/source/environment/lib/python3.7/site-packages/py4j/java_gateway.py", line 929, in _get_connection
    connection = self.deque.pop()
IndexError: pop from an empty deque

应用程序陷入了这个错误。
在主要错误发生之前，出现了一些其他错误：
在处理上述异常时，发生了另一个异常：

Traceback (most recent call last):
  File "/source/environment/lib/python3.7/site-packages/py4j/java_gateway.py", line 1067, in start
    self.socket.connect((self.address, self.port))
ConnectionRefusedError: [Errno 111] Connection refused
/source/environment/lib/python3.7/site-packages/pyspark/context.py:446: RuntimeWarning: Unable to cleanly shutdown Spark JVM process. It is possible that the process has crashed, been killed or may also be in a zombie state.
  RuntimeWarning
ERROR:py4j.java_gateway:An error occurred while trying to connect to the Java server (127.0.0.1:40277)
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/source/environment/lib/python3.7/site-packages/pyspark/sql/column.py", line 611, in between
    return (self >= lowerBound) & (self <= upperBound)
  File "/source/environment/lib/python3.7/site-packages/pyspark/sql/column.py", line 115, in _
    njc = getattr(self._jc, name)(jc)
  File "/source/environment/lib/python3.7/site-packages/py4j/java_gateway.py", line 1257, in __call__
    answer, self.gateway_client, self.target_id, self.name)
  File "/source/environment/lib/python3.7/site-packages/pyspark/sql/utils.py", line 63, in deco
    return f(*a,**kw)
  File "/source/environment/lib/python3.7/site-packages/py4j/protocol.py", line 336, in get_return_value
    format(target_id, ".", name))
py4j.protocol.Py4JError: An error occurred while calling o1081.leq

可能是因为

RuntimeWarning: Unable to cleanly shutdown Spark JVM process. It is possible that the process has crashed, been killed or may also be in a zombie state.

不知道怎么处理。