我为hadoop流运行以下命令
hadoop jar contrib/streaming/hadoop-streaming-1.0.3.jar -input test1/testregxml.xml -output testoutput2 -file ~/Programs/CMapper.php -mapper ~/Programs/CMapper.php -jobconf mapred.reduce.tasks=1
hadoop任务失败,输出如下
12/10/26 15:36:56 WARN streaming.StreamJob: -jobconf option is deprecated, please use -
D instead.
packageJobJar: [/home/HadoopNaman1/Programs/CMapper.php, /tmp/hadoop-HadoopNaman1/hadoop-unjar8119734358986860064/] [] /tmp/streamjob3840254105417044886.jar tmpDir=null
12/10/26 15:36:58 INFO util.NativeCodeLoader: Loaded the native-hadoop library
12/10/26 15:36:58 WARN snappy.LoadSnappy: Snappy native library not loaded
12/10/26 15:36:58 INFO mapred.FileInputFormat: Total input paths to process : 1
12/10/26 15:36:58 INFO streaming.StreamJob: getLocalDirs(): [/tmp/hadoop-HadoopNaman1/mapred/local]
12/10/26 15:36:58 INFO streaming.StreamJob: Running job: job_201209282004_0015
12/10/26 15:36:58 INFO streaming.StreamJob: To kill this job, run:
12/10/26 15:36:58 INFO streaming.StreamJob: /home/HadoopNaman1/hadoop-1.0.3/libexec/../bin/hadoop job -Dmapred.job.tracker=saraswati.tallysolutions.com:9051 -kill job_201209282004_0015
12/10/26 15:36:58 INFO streaming.StreamJob: Tracking URL: http://192.168.1.17:50030/jobdetails.jsp?jobid=job_201209282004_0015
12/10/26 15:36:59 INFO streaming.StreamJob: map 0% reduce 0%
12/10/26 15:37:51 INFO streaming.StreamJob: map 100% reduce 100%
12/10/26 15:37:51 INFO streaming.StreamJob: To kill this job, run:
12/10/26 15:37:51 INFO streaming.StreamJob: /home/HadoopNaman1/hadoop-1.0.3/libexec/../bin/hadoop job -Dmapred.job.tracker=saraswati.tallysolutions.com:9051 -kill job_201209282004_0015
12/10/26 15:37:51 INFO streaming.StreamJob: Tracking URL: http://192.168.1.17:50030/jobdetails.jsp?jobid=job_201209282004_0015
12/10/26 15:37:51 ERROR streaming.StreamJob: Job not successful. Error: # of failed Map Tasks exceeded allowed limit. FailedCount: 1. LastFailedTask: task_201209282004_0015_m_000001
12/10/26 15:37:51 INFO streaming.StreamJob: killJob...
Streaming Command Failed!
这是我的求职日志
2012-10-26 15:37:44,906 INFO org.apache.hadoop.mapred.TaskInProgress: TaskInProgress task_201209282004_0015_m_000001 has failed 4 times.
2012-10-26 15:37:44,906 INFO org.apache.hadoop.mapred.JobInProgress: Aborting job job_201209282004_0015
2012-10-26 15:37:44,906 INFO org.apache.hadoop.mapred.JobInProgress: Killing job 'job_201209282004_0015'
2012-10-26 15:37:44,907 INFO org.apache.hadoop.mapred.JobTracker: Adding task (JOB_CLEANUP) 'attempt_201209282004_0015_m_000002_0' to tip task_201209282004_0015_m_000002, for tracker 'tracker_192.168.1.86:localhost/127.0.0.1:45880'
2012-10-26 15:37:44,907 INFO org.apache.hadoop.mapred.JobTracker: Removing task 'attempt_201209282004_0015_m_000001_3'
2012-10-26 15:37:46,861 INFO org.apache.hadoop.mapred.TaskInProgress: Error from attempt_201209282004_0015_m_000000_3: java.lang.RuntimeException: Error in configuring object
at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:93)
at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:64)
at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:117)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:432)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:372)
at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:416)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1121)
at org.apache.hadoop.mapred.Child.main(Child.java:249)
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:616)
at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:88)
... 9 more
Caused by: java.lang.RuntimeException: Error in configuring object
at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:93)
at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:64)
at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:117)
at org.apache.hadoop.mapred.MapRunner.configure(MapRunner.java:34)
... 14 more
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:616)
at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:88)
... 17 more
Caused by: java.lang.RuntimeException: configuration exception
at org.apache.hadoop.streaming.PipeMapRed.configure(PipeMapRed.java:230)
at org.apache.hadoop.streaming.PipeMapper.configure(PipeMapper.java:66)
... 22 more
Caused by: java.io.IOException: Cannot run program "/tmp/hadoop-HadoopNaman1/mapred/local/taskTracker/HadoopNaman1/jobcache/job_201209282004_0015/attempt_201209282004_0015_m_000000_3/work/./CMapper.php": java.io.IOException: error=2, No such file or directory
at java.lang.ProcessBuilder.start(ProcessBuilder.java:475)
at org.apache.hadoop.streaming.PipeMapRed.configure(PipeMapRed.java:214)
... 23 more
Caused by: java.io.IOException: java.io.IOException: error=2, No such file or directory
at java.lang.UNIXProcess.<init>(UNIXProcess.java:164)
at java.lang.ProcessImpl.start(ProcessImpl.java:81)
at java.lang.ProcessBuilder.start(ProcessBuilder.java:468)
... 24 more
2012-10-26 15:37:49,866 INFO org.apache.hadoop.mapred.JobTracker: Removing task 'attempt_201209282004_0015_m_000000_3'
2012-10-26 15:37:50,913 INFO org.apache.hadoop.mapred.JobInProgress: Task 'attempt_201209282004_0015_m_000002_0' has completed task_201209282004_0015_m_000002 successfully.
2012-10-26 15:37:50,914 INFO org.apache.hadoop.mapred.JobInProgress$JobSummary: jobId=job_201209282004_0015,submitTime=1351246018628,launchTime=1351246018806,firstMapTaskLaunchTime=1351246028873,firstJobSetupTaskLaunchTime=1351246019863,firstJobCleanupTaskLaunchTime=1351246064906,finishTime=1351246070914,numMaps=2,numSlotsPerMap=1,numReduces=1,numSlotsPerReduce=1,user=HadoopNaman1,queue=default,status=FAILED,mapSlotSeconds=60,reduceSlotsSeconds=0,clusterMapCapacity=12,clusterReduceCapacity=12,jobName=streamjob3840254105417044886.jar
2012-10-26 15:37:50,936 INFO org.apache.hadoop.mapred.JobHistory: Moving file:/home/HadoopNaman1/hadoop-1.0.3/logs/history/job_201209282004_0015_1351246018628_HadoopNaman1_streamjob3840254105417044886.jar to file:/home/HadoopNaman1/hadoop-1.0.3/logs/history/done/version-1/saraswati.tallysolutions.com_1348842869935_/2012/10/26/000000
2012-10-26 15:37:50,942 INFO org.apache.hadoop.mapred.JobHistory: Moving file:/home/HadoopNaman1/hadoop-1.0.3/logs/history/job_201209282004_0015_conf.xml to file:/home/HadoopNaman1/hadoop-1.0.3/logs/history/done/version-1/saraswati.tallysolutions.com_1348842869935_/2012/10/26/000000
2012-10-26 15:37:50,948 INFO org.apache.hadoop.mapred.JobTracker: Removing task 'attempt_201209282004_0015_m_000002_0'
2012-10-26 15:37:50,948 INFO org.apache.hadoop.mapred.JobTracker: Removing task 'attempt_201209282004_0015_m_000003_0'
2012-10-26 15:37:51,850 INFO org.apache.hadoop.mapred.JobTracker: Killing job job_201209282004_0015
请提供建议。Map程序文件是可执行的,并且存在于目录中,权限设置为777。
1条答案
按热度按时间gpnt7bae1#
可能是在抱怨你的解释器(文件中的第一个,类似“#”/usr/bin/php)指向的解释器在该路径中不存在。hadoop会给出一些非常奇怪的/难以破译的错误,当事情在这个基本的层次上被破坏时。