spark作业在写入aws s3 bucket时失败-

nhhxz33t  于 2021-05-29  发布在  Spark
关注(0)|答案(1)|浏览(476)

spark作业在写入aws s3 bucket时失败,我得到java.io.filenotfoundexception:没有这样的文件或目录
堆栈跟踪:

java.io.FileNotFoundException: No such file or directory: s3a://vishal/test/abc.parquet/_temporary/0/task_20190422091705_0001_m_000000 
at org.apache.hadoop.fs.s3a.S3AFileSystem.getFileStatus(S3AFileSystem.java:993) 
at org.apache.hadoop.fs.s3a.S3AFileSystem.listStatus(S3AFileSystem.java:734) 
at org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter.mergePaths(FileOutputCommitter.java:360) 
at org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter.commitJob(FileOutputCommitter.java:310) 
at org.apache.parquet.hadoop.ParquetOutputCommitter.commitJob(ParquetOutputCommitter.java:48) 
at org.apache.spark.internal.io.HadoopMapReduceCommitProtocol.commitJob(HadoopMapReduceCommitProtocol.scala:166) 
at org.apache.spark.sql.execution.datasources.FileFormatWriter$.write(FileFormatWriter.scala:185) 
at org.apache.spark.sql.execution.datasources.InsertIntoHadoopFsRelationCommand.run(InsertIntoHadoopFsRelationCommand.scala:159) 
at org.apache.spark.sql.execution.command.DataWritingCommandExec.sideEffectResult$lzycompute(commands.scala:104) 
at org.apache.spark.sql.execution.command.DataWritingCommandExec.sideEffectResult(commands.scala:102) 
at org.apache.spark.sql.execution.command.DataWritingCommandExec.doExecute(commands.scala:122) 
at org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$1.apply(SparkPlan.scala:131) 
at org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$1.apply(SparkPlan.scala:127) 
at org.apache.spark.sql.execution.SparkPlan$$anonfun$executeQuery$1.apply(SparkPlan.scala:155) 
at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151) 
at org.apache.spark.sql.execution.SparkPlan.executeQuery(SparkPlan.scala:152) 
at org.apache.spark.sql.execution.SparkPlan.execute(SparkPlan.scala:127) 
at org.apache.spark.sql.execution.QueryExecution.toRdd$lzycompute(QueryExecution.scala:80) 
at org.apache.spark.sql.execution.QueryExecution.toRdd(QueryExecution.scala:80) 
at org.apache.spark.sql.DataFrameWriter$$anonfun$runCommand$1.apply(DataFrameWriter.scala:668) 
at org.apache.spark.sql.DataFrameWriter$$anonfun$runCommand$1.apply(DataFrameWriter.scala:668) 
at org.apache.spark.sql.execution.SQLExecution$$anonfun$withNewExecutionId$1.apply(SQLExecution.scala:78) 
at org.apache.spark.sql.execution.SQLExecution$.withSQLConfPropagated(SQLExecution.scala:125) 
at org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:73) 
at org.apache.spark.sql.DataFrameWriter.runCommand(DataFrameWriter.scala:668) 
at org.apache.spark.sql.DataFrameWriter.saveToV1Source(DataFrameWriter.scala:276) 
at org.apache.spark.sql.DataFrameWriter.save(DataFrameWriter.scala:270) 
at org.apache.spark.sql.DataFrameWriter.save(DataFrameWriter.scala:228) 
at org.apache.spark.sql.DataFrameWriter.parquet(DataFrameWriter.scala:557)
woobm2wo

woobm2wo1#

很可能是权限问题。从你的例外情况来看,spark job似乎没有权限 GetObject . 确保您的spark作业至少具有读、写和移动权限。
但是,如果您正在使用spark写入s3位置,并希望该文件立即可用于spark,那么它不会每次都起作用,因为
s3可能需要一些时间才能让新创建的对象出现在目录列表中,而删除的对象可能仍然可见。
如果您仍然需要这样做,我建议您在文件完成后,在可靠的持久性存储(如本地(即磁盘或hdfs))上执行所有与文件相关的操作,然后将其移动到s3。
但如果你没有其他选择,那就试试看。

相关问题