您可以尝试：

df = df.withColumn(
    'regex',
    F.regexp_extract('description', 'questionB : (.+) questionC :', 1)
)

你可以调用split函数两次，以获得想要的结果：

spark = SparkSession.builder.appName("test").getOrCreate()
data = [
    (1,
     "questionA : put returns between paragraphs questionB : indent code by 4 spaces questionC : for linebreak add 2 spaces at end"),
    (2, "questionA : add language identifier questionB : create code fences questionC : to highlight code"),
]
df = spark.createDataFrame(data, ['id', 'description'])

df.withColumn("regex", split("description", "questionB : ").getItem(1)) \
    .withColumn("regex", trim(split("regex", "question").getItem(0))).show(truncate=False)

结果：

+---+----------------------------------------------------------------------------------------------------------------------------+-----------------------+
|id |description                                                                                                                 |regex                  |
+---+----------------------------------------------------------------------------------------------------------------------------+-----------------------+
|1  |questionA : put returns between paragraphs questionB : indent code by 4 spaces questionC : for linebreak add 2 spaces at end|indent code by 4 spaces|
|2  |questionA : add language identifier questionB : create code fences questionC : to highlight code                            |create code fences     |
+---+----------------------------------------------------------------------------------------------------------------------------+-----------------------+