问题
我正在尝试安装psuedo分布式cdh而不使用cdm。一切都通过控制台“工作”。然而,当我开始使用色调时,我在尝试使用pig时收到了一个错误。
色调中显示的错误是:
ja017:无法查找与操作关联的已启动hadoop作业id[job\u local2125047777\u 0001][0000000-160112011607704-oozie-oozi-w@pig]. 失败了!
我相信这是一个错误,是由于错误的沟通,由于一个oozie工作流问题,连接Pig与作业历史服务器。
在此之前,我无法使用hue的hive,因为oozie很难在hdfs上安装oozie的sharelib。我通过在 /etc/hadoop/conf/core-site.xml
以及 /etc/oozie/conf/hadoop-conf/core-site.xml
. 正如这里所建议的:apacheoozie未能加载sharelib
脚本信息
我为在scientific linux 7上安装cdh而编写的配置脚本可在以下位置获得:https://github.com/coatless/stat490uiuc/blob/master/install_scripts/cdh_build.sh
具体来说,我试图从pig脚本中获得结果:
data = LOAD '/user/hue/pig/examples/data/midsummer.txt' as (text:CHARARRAY);
upper_case = FOREACH data GENERATE org.apache.pig.piggybank.evaluation.string.UPPER(text);
STORE upper_case INTO '$output' ;
尝试的解决方案
在google上,我遇到了以下解决方案,这些解决方案一经实施,就没有奏效。
ja017:无法查找已启动的hadoop作业id
建议运行以下命令:
sudo -u hdfs hadoop fs -mkdir -p /user/history
sudo -u hdfs hadoop fs -chmod -R 1777 /user/history
sudo -u hdfs hadoop fs -chown mapred:hadoop /user/history
重新启动资源和节点管理器、hdfs和历史服务器,但没有结果。
在线程中,有另一个用户建议在中设置属性 job.properties
说明了 user.name=mapred
. 但是,我找不到作业的job.properties的任何引用。
oozie日志报告未填充未知的hadoop作业和历史服务器ui
这篇文章建议在 mapred-site.xml
文件:
<property>
<name>mapreduce.jobhistory.done-dir</name>
<value>/user/history/done</value>
</property>
<property>
<name>mapreduce.jobhistory.intermediate-done-dir</name>
<value>/user/history/done_intermediate</value>
</property>
这也不起作用。
ja017:未知的hadoop作业
指示问题可能与权限问题有关,但是,用户没有提供有关如何解决问题的详细信息。
任何帮助都将不胜感激。
完整的oozie日志
oozie.log文件中的完整错误文本:
2016-01-11 23:51:59,195 WARN ParameterVerifier:523 - SERVER[server-name] USER[-] GROUP[-] TOKEN[-] APP[-] JOB[-] ACTION[-] The application does not define formal parameters in its XML definition
2016-01-11 23:51:59,275 WARN LiteWorkflowAppService:523 - SERVER[server-name] USER[-] GROUP[-] TOKEN[-] APP[-] JOB[-] ACTION[-] libpath [hdfs://localhost:8020/user/hue/oozie/workspaces/_cloudera_-oozie-1-1452577913.73/lib] does not exist
2016-01-11 23:51:59,572 INFO ActionStartXCommand:520 - SERVER[server-name] USER[cloudera] GROUP[-] TOKEN[] APP[pig-app-hue-script] JOB[0000000-160111235108256-oozie-oozi-W] ACTION[0000000-160111235108256-oozie-oozi-W@:start:] Start action [0000000-160111235108256-oozie-oozi-W@:start:] with user-retry state : userRetryCount [0], userRetryMax [0], userRetryInterval [10]
2016-01-11 23:51:59,595 INFO ActionStartXCommand:520 - SERVER[server-name] USER[cloudera] GROUP[-] TOKEN[] APP[pig-app-hue-script] JOB[0000000-160111235108256-oozie-oozi-W] ACTION[0000000-160111235108256-oozie-oozi-W@:start:] [***0000000-160111235108256-oozie-oozi-W@:start:***]Action status=DONE
2016-01-11 23:51:59,596 INFO ActionStartXCommand:520 - SERVER[server-name] USER[cloudera] GROUP[-] TOKEN[] APP[pig-app-hue-script] JOB[0000000-160111235108256-oozie-oozi-W] ACTION[0000000-160111235108256-oozie-oozi-W@:start:] [***0000000-160111235108256-oozie-oozi-W@:start:***]Action updated in DB!
2016-01-11 23:52:00,052 INFO ActionStartXCommand:520 - SERVER[server-name] USER[cloudera] GROUP[-] TOKEN[] APP[pig-app-hue-script] JOB[0000000-160111235108256-oozie-oozi-W] ACTION[0000000-160111235108256-oozie-oozi-W@pig] Start action [0000000-160111235108256-oozie-oozi-W@pig] with user-retry state : userRetryCount [0], userRetryMax [0], userRetryInterval [10]
2016-01-11 23:52:03,487 WARN Credentials:96 - SERVER[server-name] Null token ignored for oozie mr token
2016-01-11 23:52:03,506 WARN Credentials:96 - SERVER[server-name] Null token ignored for oozie mr token
2016-01-11 23:52:03,562 WARN JobResourceUploader:64 - SERVER[server-name] Hadoop command-line option parsing not performed. Implement the Tool interface and execute your application with ToolRunner to remedy this.
2016-01-11 23:52:03,563 WARN JobResourceUploader:171 - SERVER[server-name] No job jar file set. User classes may not be found. See Job or Job#setJar(String).
2016-01-11 23:52:04,169 WARN MRApps:582 - SERVER[server-name] cache file (mapreduce.job.cache.files) hdfs://localhost:8020/user/oozie/share/lib/lib_20160111222734/pig/json-simple-1.1.jar conflicts with cache file (mapreduce.job.cache.files) hdfs://localhost:8020/user/oozie/share/lib/lib_20160111222734/oozie/json-simple-1.1.jar This will be an error in Hadoop 2.0
2016-01-11 23:52:08,611 WARN Credentials:96 - SERVER[server-name] Null token ignored for oozie mr token
2016-01-11 23:52:08,618 WARN PigActionExecutor:523 - SERVER[server-name] USER[cloudera] GROUP[-] TOKEN[] APP[pig-app-hue-script] JOB[0000000-160111235108256-oozie-oozi-W] ACTION[0000000-160111235108256-oozie-oozi-W@pig] Exception in check(). Message[JA017: Could not lookup launched hadoop Job ID [job_local1961106749_0001] which was associated with action [0000000-160111235108256-oozie-oozi-W@pig]. Failing this action!]
org.apache.oozie.action.ActionExecutorException: JA017: Could not lookup launched hadoop Job ID [job_local1961106749_0001] which was associated with action [0000000-160111235108256-oozie-oozi-W@pig]. Failing this action!
at org.apache.oozie.action.hadoop.JavaActionExecutor.check(JavaActionExecutor.java:1274)
at org.apache.oozie.action.hadoop.JavaActionExecutor.start(JavaActionExecutor.java:1203)
at org.apache.oozie.command.wf.ActionStartXCommand.execute(ActionStartXCommand.java:250)
at org.apache.oozie.command.wf.ActionStartXCommand.execute(ActionStartXCommand.java:64)
at org.apache.oozie.command.XCommand.call(XCommand.java:286)
at org.apache.oozie.service.CallableQueueService$CompositeCallable.call(CallableQueueService.java:321)
at org.apache.oozie.service.CallableQueueService$CompositeCallable.call(CallableQueueService.java:250)
at org.apache.oozie.service.CallableQueueService$CallableWrapper.run(CallableQueueService.java:175)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
2016-01-11 23:52:08,620 WARN ActionStartXCommand:523 - SERVER[server-name] USER[cloudera] GROUP[-] TOKEN[] APP[pig-app-hue-script] JOB[0000000-160111235108256-oozie-oozi-W] ACTION[0000000-160111235108256-oozie-oozi-W@pig] Error starting action [pig]. ErrorType [FAILED], ErrorCode [JA017], Message [JA017: Could not lookup launched hadoop Job ID [job_local1961106749_0001] which was associated with action [0000000-160111235108256-oozie-oozi-W@pig]. Failing this action!]
org.apache.oozie.action.ActionExecutorException: JA017: Could not lookup launched hadoop Job ID [job_local1961106749_0001] which was associated with action [0000000-160111235108256-oozie-oozi-W@pig]. Failing this action!
at org.apache.oozie.action.hadoop.JavaActionExecutor.check(JavaActionExecutor.java:1274)
at org.apache.oozie.action.hadoop.JavaActionExecutor.start(JavaActionExecutor.java:1203)
at org.apache.oozie.command.wf.ActionStartXCommand.execute(ActionStartXCommand.java:250)
at org.apache.oozie.command.wf.ActionStartXCommand.execute(ActionStartXCommand.java:64)
at org.apache.oozie.command.XCommand.call(XCommand.java:286)
at org.apache.oozie.service.CallableQueueService$CompositeCallable.call(CallableQueueService.java:321)
at org.apache.oozie.service.CallableQueueService$CompositeCallable.call(CallableQueueService.java:250)
at org.apache.oozie.service.CallableQueueService$CallableWrapper.run(CallableQueueService.java:175)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
2016-01-11 23:52:08,621 WARN ActionStartXCommand:523 - SERVER[server-name] USER[cloudera] GROUP[-] TOKEN[] APP[pig-app-hue-script] JOB[0000000-160111235108256-oozie-oozi-W] ACTION[0000000-160111235108256-oozie-oozi-W@pig] Failing Job due to failed action [pig]
2016-01-11 23:52:08,623 WARN LiteWorkflowInstance:523 - SERVER[server-name] USER[cloudera] GROUP[-] TOKEN[] APP[pig-app-hue-script] JOB[0000000-160111235108256-oozie-oozi-W] ACTION[0000000-160111235108256-oozie-oozi-W@pig] Workflow Failed. Failing node [pig]
2016-01-11 23:52:08,768 INFO KillXCommand:520 - SERVER[server-name] USER[cloudera] GROUP[-] TOKEN[] APP[pig-app-hue-script] JOB[0000000-160111235108256-oozie-oozi-W] ACTION[] STARTED WorkflowKillXCommand for jobId=0000000-160111235108256-oozie-oozi-W
2016-01-11 23:52:08,806 INFO KillXCommand:520 - SERVER[server-name] USER[cloudera] GROUP[-] TOKEN[] APP[pig-app-hue-script] JOB[0000000-160111235108256-oozie-oozi-W] ACTION[] ENDED WorkflowKillXCommand for jobId=0000000-160111235108256-oozie-oozi-W
2016-01-11 23:52:09,038 INFO CallbackServlet:520 - SERVER[server-name] USER[-] GROUP[-] TOKEN[-] APP[-] JOB[0000000-160111235108256-oozie-oozi-W] ACTION[0000000-160111235108256-oozie-oozi-W@pig] callback for action [0000000-160111235108256-oozie-oozi-W@pig]
2016-01-11 23:52:09,072 ERROR CompletedActionXCommand:517 - SERVER[server-name] USER[-] GROUP[-] TOKEN[] APP[-] JOB[0000000-160111235108256-oozie-oozi-W] ACTION[0000000-160111235108256-oozie-oozi-W@pig] XException,
org.apache.oozie.command.CommandException: E0800: Action it is not running its in [FAILED] state, action [0000000-160111235108256-oozie-oozi-W@pig]
at org.apache.oozie.command.wf.CompletedActionXCommand.eagerVerifyPrecondition(CompletedActionXCommand.java:92)
at org.apache.oozie.command.XCommand.call(XCommand.java:257)
at org.apache.oozie.service.CallableQueueService$CallableWrapper.run(CallableQueueService.java:175)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
2016-01-11 23:52:09,082 WARN CallableQueueService$CallableWrapper:523 - SERVER[server-name] USER[-] GROUP[-] TOKEN[] APP[-] JOB[0000000-160111235108256-oozie-oozi-W] ACTION[0000000-160111235108256-oozie-oozi-W@pig] exception callable [callback], E0800: Action it is not running its in [FAILED] state, action [0000000-160111235108256-oozie-oozi-W@pig]
org.apache.oozie.command.CommandException: E0800: Action it is not running its in [FAILED] state, action [0000000-160111235108256-oozie-oozi-W@pig]
at org.apache.oozie.command.wf.CompletedActionXCommand.eagerVerifyPrecondition(CompletedActionXCommand.java:92)
at org.apache.oozie.command.XCommand.call(XCommand.java:257)
at org.apache.oozie.service.CallableQueueService$CallableWrapper.run(CallableQueueService.java:175)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
1条答案
按热度按时间4uqofj5v1#
应该使用hue文件浏览器仔细检查/user/history的所有目录和子目录上的所有权限是否正确。
在我的例子中,所有用户对/user/history的所有子文件夹都有权限,但文件浏览器告诉我,“/user/history”目录本身有以下权限集:
这导致使用不同于mapred的用户时出错。以下命令有帮助: