apache crunch错误

dkqlctbz  于 2021-06-03  发布在  Hadoop
关注(0)|答案(1)|浏览(298)

我有一个读取*.avro文件并加载到loadavro*.avro tablename表的命令
当我尝试在vagrant集群中运行代码时,它抛出以下错误

[root@cdh4-cluster vagrant]# hadoop jar devcenter-store-load-0.2-SNAPSHOT.jar loadavro person.avro person_table

Unexpected exception: Cannot create job output directory /tmp/crunch-1880716375
java.lang.RuntimeException: Cannot create job output directory /tmp/crunch-1880716375
        at org.apache.crunch.impl.dist.DistributedPipeline.createTempDirectory(DistributedPipeline.java:281)
        at org.apache.crunch.impl.dist.DistributedPipeline.<init>(DistributedPipeline.java:88)
        at org.apache.crunch.impl.mr.MRPipeline.<init>(MRPipeline.java:88)
        at org.apache.crunch.impl.mr.MRPipeline.<init>(MRPipeline.java:76)
        at com.cerner.devcenter.pipeline.crunch.PersonRecordPipeline.getPipelineInstance(PersonRecordPipeline.java:72)
        at com.cerner.devcenter.pipeline.crunch.PersonRecordPipeline.writeIntoHBase(PersonRecordPipeline.java:53)
        at com.cerner.console.commands.LoadAvroCommand.run(LoadAvroCommand.java:35)
        at com.cerner.kepler.commands.Command.run(Command.java:112)
        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
        at com.cerner.kepler.commands.BasicCommandRunner.run(BasicCommandRunner.java:61)
        at com.cerner.kepler.commands.CommandRunner.run(CommandRunner.java:19)
        at com.cerner.console.main.ConsoleMain.main(ConsoleMain.java:26)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at org.apache.hadoop.util.RunJar.main(RunJar.java:208)
Caused by: org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.hdfs.server.namenode.SafeModeException): Cannot create directory /tmp/crunch-1880716375. Name node is in safe mode.
The reported blocks 104 needs additional 2 blocks to reach the threshold 0.9990 of total blocks 106. Safe mode will be turned off automatically.
        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.mkdirsInternal(FSNamesystem.java:2982)
        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.mkdirsInt(FSNamesystem.java:2960)
        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.mkdirs(FSNamesystem.java:2938)
        at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.mkdirs(NameNodeRpcServer.java:648)
        at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.mkdirs(ClientNamenodeProtocolServerSideTranslatorPB.java:417)
        at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java:44096)
        at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:453)
        at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1002)
        at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1695)
        at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1691)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:396)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1408)
        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1689)
        at org.apache.hadoop.ipc.Client.call(Client.java:1225)
        at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:202)
        at $Proxy10.mkdirs(Unknown Source)
        at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.mkdirs(ClientNamenodeProtocolTranslatorPB.java:425)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:164)
        at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:83)
        at $Proxy11.mkdirs(Unknown Source)
        at org.apache.hadoop.hdfs.DFSClient.primitiveMkdir(DFSClient.java:2121)
        at org.apache.hadoop.hdfs.DFSClient.mkdirs(DFSClient.java:2092)
        at org.apache.hadoop.hdfs.DistributedFileSystem.mkdirs(DistributedFileSystem.java:546)
        at org.apache.hadoop.fs.FileSystem.mkdirs(FileSystem.java:1902)
        at org.apache.crunch.impl.dist.DistributedPipeline.createTempDirectory(DistributedPipeline.java:279)
        ... 16 more
0ve6wy6x

0ve6wy6x1#

如果查看异常,您可以看到namenode处于安全模式,因此crunch作业无法将任何新文件写入hdfs。将namenode移出安全模式,然后再次运行作业。hadoop dfsadmin-安全模式离开

相关问题