我已经编写了scalaspark代码来构建我的项目,ide是intellij,它在aws emr集群上运行时显示了这个错误,并且在本地服务器上运行良好。
它在下面一行开裂:
var join_sql="select ipfile.id,ipfile.col1,opfile.col2 from ipfile join opfile on ipfile.id=opfile.id"
var df1=Operation.spark.sql(join_sql)
df1.createOrReplaceTempView("df1")
var df2 = df1.groupBy("col1","col2").count()
df2.createOrReplaceTempView("df2")
df2=Operation.spark.sql("select * from df2 order by count desc")
print("count : ",df2.count())
try {
df2.foreach(t => {
impact=t.getAs[Long]("impact").toString // Job was aborting at this particular line
m1 = t.getAs[String]("col1")
m2=t.getAs[String]("col2")
print("m1" + "m2" )
})
当我通过sbt assembly创建jar以在本地模式下运行它时,它工作正常,但是当我为yarn客户机创建jar并在集群模式下执行它时,它显示了这个错误。
暂无答案!
目前还没有任何答案,快来回答吧!