请允许我从两个简短的观察开始：
我相信避免用单个数字来命名我们的列会更安全。想想我们需要计算表达式的情况 1 is not null . 在这里，我们的意思是 column 1 或者 value 1 它自己。
据我所知，通过Dataframe存储和处理目标列是不可行的。这将创建一个开销，可以通过使用单个scala集合（即seq、array、set等）轻松避免。
下面是解决问题的方法：

import org.apache.spark.sql.functions.col

val id_df = Seq(
  ("c1","gender"),
  ("c2","city"),
  ("c3","state"),
  ("c4","age")
).toDF("id","type")

val main_df = Seq(
    ("male", "los angeles", null),
    ("female", "new york", "new york"),
    ("trans", null, "new york")
).toDF("c1","c2","c3")

val targetCols = id_df.collect()
                      .map{_.getString(0)} //get id
                      .toSet //convert current sequence to a set (required for the intersection)
                      .intersect(main_df.columns.toSet) //get common columns with main_df
                      .map(col(_).isNotNull) //convert c1,..cN to col(c[i]).isNotNull
                      .reduce(_ && _) // apply the AND operator between items

// (((c1 IS NOT NULL) AND (c2 IS NOT NULL)) AND (c3 IS NOT NULL))

main_df.withColumn("meets_conditions", targetCols).show(false)

// +------+-----------+--------+----------------+
// |c1    |c2         |c3      |meets_conditions|
// +------+-----------+--------+----------------+
// |male  |los angeles|null    |false           |
// |female|new york   |new york|true            |
// |trans |null       |new york|false           |
// +------+-----------+--------+----------------+