执行使用并集和秩的pig脚本(错误2017:创建作业配置时发生内部错误)

enyaitl3  于 2021-05-29  发布在  Hadoop
关注(0)|答案(2)|浏览(558)

我执行上面显示的简单pig脚本,它使用秩操作符和并集。它抛出:error 2017:internal error creating job configuration。

A = LOAD '/location/to/my/file';
SPLIT A INTO 
B IF $0 IS NULL,
C IF $0 IS NOT NULL;

B2 = RANK B;
C2 = RANK C;

D = UNION B2, C2;
STORE D INTO '/location/to/my/outputs/1';

有人知道怎么解决这个问题吗?
完整的错误日志是:

org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobCreationException: ERROR 2017: Internal error creating job configuration.
  at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler.getJob(JobControlCompiler.java:873)
  at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler.compile(JobControlCompiler.java:298)
  at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher.launchPig(MapReduceLauncher.java:190)
  at org.apache.pig.PigServer.launchPlan(PigServer.java:1334)
  at org.apache.pig.PigServer.executeCompiledLogicalPlan(PigServer.java:1319)
  at org.apache.pig.PigServer.execute(PigServer.java:1309)
  at org.apache.pig.PigServer.executeBatch(PigServer.java:387)
  at org.apache.pig.PigServer.executeBatch(PigServer.java:365)
  at org.apache.pig.tools.grunt.GruntParser.executeBatch(GruntParser.java:140)
  at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:202)
  at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:173)
  at org.apache.pig.tools.grunt.Grunt.exec(Grunt.java:84)
  at org.apache.pig.Main.run(Main.java:478)
  at org.apache.pig.PigRunner.run(PigRunner.java:49)
  at org.apache.oozie.action.hadoop.PigMain.runPigJob(PigMain.java:285)
  at org.apache.oozie.action.hadoop.PigMain.run(PigMain.java:228)
  at org.apache.oozie.action.hadoop.LauncherMain.run(LauncherMain.java:46)
  at org.apache.oozie.action.hadoop.PigMain.main(PigMain.java:76)
  at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
  at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
  at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
  at java.lang.reflect.Method.invoke(Method.java:606)
  at org.apache.oozie.action.hadoop.LauncherMapper.map(LauncherMapper.java:228)
  at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
  at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:453)
  at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
  at org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler.runSubtask(LocalContainerLauncher.java:370)
  at org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler.runTask(LocalContainerLauncher.java:295)
  at org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler.access$200(LocalContainerLauncher.java:181)
  at org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler$1.run(LocalContainerLauncher.java:224)
  at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
  at java.util.concurrent.FutureTask.run(FutureTask.java:262)
  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
  at java.lang.Thread.run(Thread.java:745)
  Caused by: java.lang.NullPointerException
  at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler.getJob(JobControlCompiler.java:817)
  ... 34 more
eni9jsuy

eni9jsuy1#

这个问题已经在上面的链接中展开:https://issues.apache.org/jira/browse/pig-3985
它似乎在pig版本0.14.0上得到了解决
不幸的是,我正在开发0.12pig版本,它需要一个特定的补丁来修复cloudera的问题。

az31mfrm

az31mfrm2#

这会有用的。。只要写一个foreach后,你申请职级,然后联合的关系

A = LOAD '/location/to/my/file';
SPLIT A INTO 
B IF $0 IS NULL,
C IF $0 IS NOT NULL;

B2 = RANK B;
C2 = RANK C;

B_EACH = FOREACH B2 *;
C_EACH = FOREACH C2 *;

D = UNION B_EACH,C_EACH;
STORE D INTO '/location/to/my/outputs/1';

相关问题