在计算机上本地工作时,我无法抑制pyspark警告。我收到以下警告,我想“抑制”它们:
WARN Utils: Your hostname, [HOSTNAME] resolves to a loopback address: 127.0.1.1; using 192.168.26.41 instead (on interface [INTERFACE])
WARN Utils: Set SPARK_LOCAL_IP if you need to bind to another address
Setting default log level to "WARN".
To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).
WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
字符串
以下是我试图抑制警告的内容,但它们都不起作用:
from pyspark.sql import SparkSession
method_number = 1 # 2, 3
def get_spark_object():
return SparkSession.builder.getOrCreate()
if method_number == 1:
import sys
save_stdout = sys.stdout
sys.stdout = open('trash', 'w')
spark = get_spark_object()
sys.stdout = save_stdout
elif method_number == 2:
import warnings
warnings.filterwarnings('ignore')
warnings.simplefilter('ignore')
spark = get_spark_object()
elif method_number == 3:
import shutup
shutup.please()
spark = get_spark_object()
型
1条答案
按热度按时间tez616oj1#
这里有一个替代方法,可以帮助抑制这些特定的警告。
字符串
这样,您就不需要重定向或过滤警告;相反,您正在配置Spark以避免发出某些警告。