pig输出位置验证失败

bq3bfh9z  于 2021-06-03  发布在  Hadoop
关注(0)|答案(1)|浏览(367)

在执行下面给定的脚本时,我得到如下所述的错误。我尝试了现有的目录以及不存在的目录,但我仍然得到相同的错误。因为我刚开始写Pig的剧本,请帮我解决这个问题。我正在尝试加载文件并在应用snappy转换后将其存储在不同的位置。
代码

SET output.compression.enabled true;
SET mapred.map.output.compression.codec org.apache.hadoop.io.compress.SnappyCodec;
A = LOAD '/abc/def/' using PigStorage();
Store A into '/abc/def/snappy/' using PigStorage();

错误

Pig Stack Trace
---------------
ERROR 2116:
<file snappy_conversion.pig, line 4, column 0> Output Location Validation Failed for: '/abc/def/snappy/

org.apache.pig.impl.plan.VisitorException: ERROR 2116:
<file snappy_conversion.pig, line 4, column 0> Output Location Validation Failed for: '/abc/def/snappy/
        at org.apache.pig.newplan.logical.rules.InputOutputFileValidator$InputOutputFileVisitor.visit(InputOutputFileValidator.java:75)
        at org.apache.pig.newplan.logical.relational.LOStore.accept(LOStore.java:66)
        at org.apache.pig.newplan.DepthFirstWalker.depthFirst(DepthFirstWalker.java:64)
        at org.apache.pig.newplan.DepthFirstWalker.depthFirst(DepthFirstWalker.java:66)
        at org.apache.pig.newplan.DepthFirstWalker.walk(DepthFirstWalker.java:53)
        at org.apache.pig.newplan.PlanVisitor.visit(PlanVisitor.java:52)
        at org.apache.pig.newplan.logical.rules.InputOutputFileValidator.validate(InputOutputFileValidator.java:45)
        at org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.compile(HExecutionEngine.java:285)
        at org.apache.pig.PigServer.compilePp(PigServer.java:1324)
        at org.apache.pig.PigServer.executeCompiledLogicalPlan(PigServer.java:1249)
        at org.apache.pig.PigServer.execute(PigServer.java:1241)
        at org.apache.pig.PigServer.executeBatch(PigServer.java:335)
        at org.apache.pig.tools.grunt.GruntParser.executeBatch(GruntParser.java:137)
        at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:198)
        at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:170)
        at org.apache.pig.tools.grunt.Grunt.exec(Grunt.java:84)
        at org.apache.pig.Main.run(Main.java:604)
        at org.apache.pig.Main.main(Main.java:157)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at org.apache.hadoop.util.RunJar.main(RunJar.java:208)
Caused by: java.lang.NullPointerException
        at java.lang.Class.forName0(Native Method)
        at java.lang.Class.forName(Class.java:169)
        at org.apache.pig.builtin.PigStorage.setStoreLocation(PigStorage.java:396)
        at org.apache.pig.newplan.logical.rules.InputOutputFileValidator$InputOutputFileVisitor.visit(InputOutputFileValidator.java:68)
        ... 22 more
================================================================================
vngu2lb8

vngu2lb81#

问题是您设置了错误的配置。正确的设置是

SET output.compression.codec org.apache.hadoop.io.compress.SnappyCodec;
``` `mapred.map.output.compression.codec` 用于设置Map器输出的中间文件的压缩格式。
您可以阅读此处的代码以获取更多信息:http://www.grepcode.com/file/repo1.maven.org/maven2/org.apache.pig/pig/0.12.0/org/apache/pig/builtin/pigstorage.java?av=h#440

相关问题