如果你检查colrefex方法的Spark码。。。它希望正则表达式以以下格式传递

/**the column name pattern in quoted regex without qualifier */
 val escapedIdentifier = "`(.+)`".r
 /**the column name pattern in quoted regex with qualifier */
 val qualifiedEscapedIdentifier = ("(.+)" + """.""" + "`(.+)`").r

backticks（`）是封闭regex所必需的，否则上面的模式将无法识别您的输入模式。
您可以尝试选择如下所述有效的特定列

val df = Seq(("id","a_in","a_out","b_in","b_out"))
  .toDF("id","a_in","a_out","b_in","b_out")

val df_in = df
  .withColumnRenamed("a_in","a")
  .withColumnRenamed("b_in","b")
  .drop(df.colRegex("`.*_(in|out)`"))
val validColumns = df_in.columns.filter(p => p.matches(".*_(in|out)$")).toSeq //select all junk columns
val final_df_in = df_in.drop(validColumns:_*) // this will drop all columns which are not valid as per your criteria.

除了waqar ahmed和kavetiraviteja（公认答案）提出的解决方案之外，还有另一种基于 select 用一些负正则表达式魔法。更简洁，但更难阅读非正则表达式大师。。。

val df_in = df
  .withColumnRenamed("a_in","a")
  .withColumnRenamed("b_in","b")
  .select(df.colRegex("`^(?!.*_(in|out)_).*$`")) // regex with negative lookahead