spark无法读取本地文件

insrf1ej  于 2021-06-02  发布在  Hadoop
关注(0)|答案(1)|浏览(469)

我在emr群集中的所有spark节点上都有一个本地文件,具有以下权限:

-rw-rw---- 1 test_user test_group 30 Jun 21 14:20 /tmp/foo_test

我正在运行集群 ec2-user ,使用Yarn调度程序。为了让spark/yarn能够访问该文件,我添加了 test_group 作为第二组 yarn 所有节点上的用户。

$ sudo -u yarn groups
  yarn hadoop test_group

在spark shell中,读取文件时出现以下错误:

scala> val rdd = sc.textFile("file:///tmp/foo_test")
    org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 0.0 failed 4 times, most recent failure: Lost task 0.3 in stage 0.0:
java.io.FileNotFoundException: /tmp/foo_test (Permission denied)
    at java.io.FileInputStream.open(Native Method)
    at java.io.FileInputStream.<init>(FileInputStream.java:146)
    at org.apache.hadoop.fs.RawLocalFileSystem$LocalFSFileInputStream.<init>(RawLocalFileSystem.java:111)
    at org.apache.hadoop.fs.RawLocalFileSystem.open(RawLocalFileSystem.java:207)
    at org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSInputChecker.<init>(ChecksumFileSystem.java:141)
    at org.apache.hadoop.fs.ChecksumFileSystem.open(ChecksumFileSystem.java:341)
    at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:771)
    at org.apache.hadoop.mapred.LineRecordReader.<init>(LineRecordReader.java:109)
    at org.apache.hadoop.mapred.TextInputFormat.getRecordReader(TextInputFormat.java:67)
    at org.apache.spark.rdd.HadoopRDD$$anon$1.<init>(HadoopRDD.scala:237)
    at org.apache.spark.rdd.HadoopRDD.compute(HadoopRDD.scala:208)
    at org.apache.spark.rdd.HadoopRDD.compute(HadoopRDD.scala:101)
    at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:306)
    at org.apache.spark.rdd.RDD.iterator(RDD.scala:270)
    at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
    at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:306)
    at org.apache.spark.rdd.RDD.iterator(RDD.scala:270)
    at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:66)
    at org.apache.spark.scheduler.Task.run(Task.scala:89)
    at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:214)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:745)

如何读取emr spark上具有组级权限的文件?

csbfibhn

csbfibhn1#

将权限更改为770-即。, chmod -R 770 /tmp 然后重试。

相关问题