scala—将行值从spark df传递到函数

4urapxun  于 2021-05-19  发布在  Spark
关注(0)|答案(1)|浏览(498)
val someDF = Seq(
  (8, "abc"),
  (64, "sdf"),
  (-27, "yui")
).toDF("number", "word")

def numberValidation(numberValidation:Any): Boolean=
numberValidation match{
case int :Integer => true
case _ => false}

def wordValidatoin(wordValidation:Any): Boolean=
wordValidation match{
case str: String => true
case _ => false}

我需要将每一行及其coresponding值传递给上述函数。第一行数字列的值=8,传递给numbervalidation,第一行单词列的值传递给wordvalidatoin

wfveoks0

wfveoks01#

使用spark自定义项并按以下方式更改方法:

val numberValidation = udf((numberValidation: Any) => {
     numberValidation match{
    case int :Integer => true
    case _ => false
    }
  })

val wordValidatoin = udf((wordValidation:Any)): =>{
    wordValidation match{
    case str: String => true
    case _ => false
    }
})

并与df核对如下:

someDF
.withColumn("checkNum",numberValidation($"number"))
.withColumn("checWord",wordValidatoin($"word"))
.show

相关问题