在pyspark版本3.5.0中，当我在批处理和流处理版本中指定StringType()时，我没有发现数字被空。

from pyspark.sql import Row
from pyspark.sql.types import *
from pyspark import SparkContext, SQLContext
from pyspark.sql.functions import *
import sys
from pyspark.sql.functions import when, max, greatest

sc = SparkContext('local')
sqlContext = SQLContext(sc)

inputFile = "../data/unionDataTypeDir"
schema_provided = StructType([StructField("value", StringType(), nullable=True)])

initialDF = sqlContext.read.schema(schema_provided).json(inputFile)

print("pyspark version :", sc.version)

print("Normal batch dataframe")
initialDF.show(n=100, truncate=False)

streaming_df = sqlContext.readStream \
    .schema(schema_provided) \
    .json(inputFile)

print("Streaming dataframe")
query = streaming_df.writeStream \
    .outputMode("append") \
    .format("console") \
    .start()

query.awaitTermination()

字符串
输出量：

pyspark version : 3.5.0
Normal batch dataframe
+-----+
|value|
+-----+
|123  |
|456  |
|NULL |
+-----+

Streaming dataframe
-------------------------------------------
Batch: 0
-------------------------------------------
+-----+
|value|
+-----+
|  123|
|  456|
| NULL|
+-----+

型