java—启用hadoop调度器(资源感知自适应调度器)

px9o7tmv  于 2021-06-03  发布在  Hadoop
关注(0)|答案(1)|浏览(305)

我想在hadoop 0.20.203.0中启用adaptivescheduler。我有一个来自这个调度程序的jar文件(我确信这个jar文件工作正常)。我将jar文件放在hadoop\u home/lib中,并在hadoop-env.sh中设置hadoop\u类路径。我在mapred-site.xml中设置了调度器的必需属性。当我运行集群时,所有的jobtracker,datanode,。。。。开始。但当我进入调度程序的ui时( http://localhost:50030/scheduler ),我面临错误404。这个调度器的jar文件包含hadoop-0.20.203.0的核心。我该怎么解决这个问题?请帮帮我。我的jobtracker日志是:

2013-07-27 01:22:29,333 INFO org.apache.hadoop.mapred.JobTracker: STARTUP_MSG: 
/************************************************************
STARTUP_MSG: Starting JobTracker
STARTUP_MSG:   host = master/192.168.0.112
STARTUP_MSG:   args = []
STARTUP_MSG:   version = 0.20.203.0
STARTUP_MSG:   build = http://svn.apache.org/repos/asf/hadoop/common/branches/branch-0.20-security-203 -r 1099333; compiled by 'oom' on Wed May  4 07:57:50 PDT 2011

************************************************************/

2013-07-27 01:22:29,527 INFO org.apache.hadoop.metrics2.impl.MetricsConfig: loaded properties from hadoop-metrics2.properties
2013-07-27 01:22:29,537 INFO org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source MetricsSystem,sub=Stats registered.
2013-07-27 01:22:29,538 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled snapshot period at 10 second(s).
2013-07-27 01:22:29,538 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: JobTracker metrics system started
2013-07-27 01:22:29,781 INFO org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source ugi registered.
2013-07-27 01:22:29,784 WARN org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Source name ugi already exists!
2013-07-27 01:22:29,785 INFO org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager: Updating the current master key for generating delegation tokens
2013-07-27 01:22:29,796 INFO org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager: Starting expired delegation token remover thread, tokenRemoverScanInterval=60 min(s)
2013-07-27 01:22:29,796 INFO org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager: Updating the current master key for generating delegation tokens
2013-07-27 01:22:29,796 INFO org.apache.hadoop.mapred.JobTracker: Scheduler configured with (memSizeForMapSlotOnJT, memSizeForReduceSlotOnJT, limitMaxMemForMapTasks, limitMaxMemForReduceTasks) (-1, -1, -1, -1)
2013-07-27 01:22:29,797 INFO org.apache.hadoop.util.HostsFileReader: Refreshing hosts (include/exclude) list
2013-07-27 01:22:29,827 INFO org.apache.hadoop.mapred.JobTracker: Starting jobtracker with owner as maedeh
2013-07-27 01:22:29,852 INFO org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source RpcDetailedActivityForPort9001 registered.
2013-07-27 01:22:29,853 INFO org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source RpcActivityForPort9001 registered.
2013-07-27 01:22:29,856 INFO org.apache.hadoop.ipc.Server: Starting SocketReader
2013-07-27 01:22:35,276 INFO org.mortbay.log: Logging to org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog
2013-07-27 01:22:35,404 INFO org.apache.hadoop.http.HttpServer: Added global filtersafety (class=org.apache.hadoop.http.HttpServer$QuotingInputFilter)
2013-07-27 01:22:35,668 INFO org.apache.hadoop.http.HttpServer: Port returned by webServer.getConnectors()[0].getLocalPort() before open() is -1. Opening the listener on 50030
2013-07-27 01:22:35,669 INFO org.apache.hadoop.http.HttpServer: listener.getLocalPort() returned 50030 webServer.getConnectors()[0].getLocalPort() returned 50030
2013-07-27 01:22:35,669 INFO org.apache.hadoop.http.HttpServer: Jetty bound to port 50030
2013-07-27 01:22:35,669 INFO org.mortbay.log: jetty-6.1.x
2013-07-27 01:22:36,225 INFO org.mortbay.log: Started SelectChannelConnector@0.0.0.0:50030
2013-07-27 01:22:36,233 INFO org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source jvm registered.
2013-07-27 01:22:36,234 INFO org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source JobTrackerMetrics registered.
2013-07-27 01:22:36,234 INFO org.apache.hadoop.mapred.JobTracker: JobTracker up at: 9001
2013-07-27 01:22:36,234 INFO org.apache.hadoop.mapred.JobTracker: JobTracker webserver: 50030
2013-07-27 01:22:36,366 INFO org.apache.hadoop.mapred.JobTracker: Cleaning up the system directory
2013-07-27 01:22:36,468 INFO org.apache.hadoop.mapred.JobHistory: Creating DONE folder at file:/home/maedeh/hadoop-0.20.203.0/logs/history/done
2013-07-27 01:22:36,478 INFO org.apache.hadoop.mapred.JobTracker: History server being initialized in embedded mode
2013-07-27 01:22:36,481 INFO org.apache.hadoop.mapred.JobHistoryServer: Started job history server at: localhost:50030
2013-07-27 01:22:36,481 INFO org.apache.hadoop.mapred.JobTracker: Job History Server web address: localhost:50030
2013-07-27 01:22:36,484 INFO org.apache.hadoop.mapred.CompletedJobStatusStore: Completed job store is inactive
2013-07-27 01:22:36,782 INFO org.apache.hadoop.mapred.AdaptiveScheduler: Successfully configured AdaptiveScheduler
2013-07-27 01:22:36,782 INFO org.apache.hadoop.mapred.JobTracker: Refreshing hosts information
2013-07-27 01:22:36,791 INFO org.apache.hadoop.util.HostsFileReader: Setting the includes file to 
2013-07-27 01:22:36,791 INFO org.apache.hadoop.util.HostsFileReader: Setting the excludes file to 
2013-07-27 01:22:36,791 INFO org.apache.hadoop.util.HostsFileReader: Refreshing hosts (include/exclude) list
2013-07-27 01:22:36,791 INFO org.apache.hadoop.mapred.JobTracker: Decommissioning 0 nodes
2013-07-27 01:22:36,802 INFO org.apache.hadoop.ipc.Server: IPC Server Responder: starting
2013-07-27 01:22:36,802 INFO org.apache.hadoop.ipc.Server: IPC Server listener on 9001: starting
2013-07-27 01:22:36,803 INFO org.apache.hadoop.ipc.Server: IPC Server handler 0 on 9001: starting
2013-07-27 01:22:36,804 INFO org.apache.hadoop.ipc.Server: IPC Server handler 2 on 9001: starting
2013-07-27 01:22:36,804 INFO org.apache.hadoop.ipc.Server: IPC Server handler 1 on 9001: starting
2013-07-27 01:22:36,804 INFO org.apache.hadoop.ipc.Server: IPC Server handler 3 on 9001: starting
2013-07-27 01:22:36,804 INFO org.apache.hadoop.ipc.Server: IPC Server handler 4 on 9001: starting
2013-07-27 01:22:36,805 INFO org.apache.hadoop.ipc.Server: IPC Server handler 5 on 9001: starting
2013-07-27 01:22:36,805 INFO org.apache.hadoop.ipc.Server: IPC Server handler 6 on 9001: starting
2013-07-27 01:22:36,805 INFO org.apache.hadoop.ipc.Server: IPC Server handler 7 on 9001: starting
2013-07-27 01:22:36,805 INFO org.apache.hadoop.ipc.Server: IPC Server handler 8 on 9001: starting
2013-07-27 01:22:36,806 INFO org.apache.hadoop.mapred.JobTracker: Starting RUNNING
2013-07-27 01:22:36,806 INFO org.apache.hadoop.ipc.Server: IPC Server handler 9 on 9001: starting
2013-07-27 01:22:46,806 INFO org.apache.hadoop.net.NetworkTopology: Adding a new node: /default-rack/slave1
2013-07-27 01:22:46,808 INFO org.apache.hadoop.mapred.JobTracker: Adding tracker tracker_slave1:localhost/127.0.0.1:58226 to host slave1
2013-07-27 01:22:47,856 INFO org.apache.hadoop.net.NetworkTopology: Adding a new node: /default-rack/slave2
2013-07-27 01:22:47,859 INFO org.apache.hadoop.mapred.JobTracker: Adding tracker tracker_slave2:localhost/127.0.0.1:55061 to host slave2
2013-07-27 01:26:28,522 INFO org.apache.hadoop.mapred.JobInProgress: job_201307270122_0001: nMaps=3 nReduces=1 max=-1
2013-07-27 01:26:28,525 INFO org.apache.hadoop.mapred.JobTracker: Job job_201307270122_0001 added successfully for user 'maedeh' to queue 'default'
2013-07-27 01:26:28,538 INFO org.apache.hadoop.mapred.AuditLogger: USER=maedeh  IP=192.168.0.112    OPERATION=SUBMIT_JOB    TARGET=job_201307270122_0001    RESULT=SUCCESS
2013-07-27 01:26:28,560 INFO org.apache.hadoop.mapred.JobTracker: Initializing job_201307270122_0001
2013-07-27 01:26:28,560 INFO org.apache.hadoop.mapred.JobInProgress: Initializing job_201307270122_0001
2013-07-27 01:26:29,359 INFO org.apache.hadoop.mapred.JobInProgress: jobToken generated and stored with users keys in /home/maedeh/tempdir/mapred/system/job_201307270122_0001/jobToken
2013-07-27 01:26:29,403 INFO org.apache.hadoop.mapred.JobInProgress: Input size for job job_201307270122_0001 = 3671523. Number of splits = 3
2013-07-27 01:26:29,404 INFO org.apache.hadoop.mapred.JobInProgress: tip:task_201307270122_0001_m_000000 has split on node:/default-rack/slave1
2013-07-27 01:26:29,404 INFO org.apache.hadoop.mapred.JobInProgress: tip:task_201307270122_0001_m_000000 has split on node:/default-rack/slave2
2013-07-27 01:26:29,404 INFO org.apache.hadoop.mapred.JobInProgress: tip:task_201307270122_0001_m_000001 has split on node:/default-rack/slave1
2013-07-27 01:26:29,405 INFO org.apache.hadoop.mapred.JobInProgress: tip:task_201307270122_0001_m_000001 has split on node:/default-rack/slave2
2013-07-27 01:26:29,405 INFO org.apache.hadoop.mapred.JobInProgress: tip:task_201307270122_0001_m_000002 has split on node:/default-rack/slave1
2013-07-27 01:26:29,405 INFO org.apache.hadoop.mapred.JobInProgress: tip:task_201307270122_0001_m_000002 has split on node:/default-rack/slave2
2013-07-27 01:26:29,405 INFO org.apache.hadoop.mapred.JobInProgress: job_201307270122_0001 LOCALITY_WAIT_FACTOR=1.0
2013-07-27 01:26:29,405 INFO org.apache.hadoop.mapred.JobInProgress: Job job_201307270122_0001 initialized successfully with 3 map tasks and 1 reduce tasks.
2013-07-27 01:26:29,708 INFO org.apache.hadoop.mapred.JobTracker: Adding task (JOB_SETUP) 'attempt_201307270122_0001_m_000004_0' to tip task_201307270122_0001_m_000004, for tracker 'tracker_slave1:localhost/127.0.0.1:58226'
2013-07-27 01:26:39,051 INFO org.apache.hadoop.mapred.JobInProgress: Task 'attempt_201307270122_0001_m_000004_0' has completed task_201307270122_0001_m_000004 successfully.
2013-07-27 01:26:39,069 INFO org.apache.hadoop.mapred.JobTracker: Adding task (MAP) 'attempt_201307270122_0001_m_000000_0' to tip task_201307270122_0001_m_000000, for tracker 'tracker_slave1:localhost/127.0.0.1:58226'
2013-07-27 01:26:39,073 INFO org.apache.hadoop.mapred.JobInProgress: Choosing data-local task task_201307270122_0001_m_000000
2013-07-27 01:26:40,326 INFO org.apache.hadoop.mapred.JobTracker: Adding task (MAP) 'attempt_201307270122_0001_m_000001_0' to tip task_201307270122_0001_m_000001, for tracker 'tracker_slave2:localhost/127.0.0.1:55061'
2013-07-27 01:26:40,345 INFO org.apache.hadoop.mapred.JobInProgress: Choosing data-local task task_201307270122_0001_m_000001
2013-07-27 01:26:42,214 INFO org.apache.hadoop.mapred.JobTracker: Adding task (MAP) 'attempt_201307270122_0001_m_000002_0' to tip task_201307270122_0001_m_000002, for tracker 'tracker_slave1:localhost/127.0.0.1:58226'
2013-07-27 01:26:42,214 INFO org.apache.hadoop.mapred.JobInProgress: Choosing data-local task task_201307270122_0001_m_000002
2013-07-27 01:27:00,452 INFO org.apache.hadoop.mapred.JobInProgress: Task 'attempt_201307270122_0001_m_000000_0' has completed task_201307270122_0001_m_000000 successfully.
2013-07-27 01:27:01,759 INFO org.apache.hadoop.mapred.JobInProgress: Task 'attempt_201307270122_0001_m_000001_0' has completed task_201307270122_0001_m_000001 successfully.
2013-07-27 01:27:06,476 INFO org.apache.hadoop.mapred.JobInProgress: Task 'attempt_201307270122_0001_m_000002_0' has completed task_201307270122_0001_m_000002 successfully.
2013-07-27 01:27:09,536 INFO org.apache.hadoop.mapred.JobTracker: Adding task (REDUCE) 'attempt_201307270122_0001_r_000000_0' to tip task_201307270122_0001_r_000000, for tracker 'tracker_slave1:localhost/127.0.0.1:58226'
2013-07-27 01:27:21,749 INFO org.apache.hadoop.mapred.JobInProgress: Task 'attempt_201307270122_0001_r_000000_0' has completed task_201307270122_0001_r_000000 successfully.
2013-07-27 01:27:21,756 INFO org.apache.hadoop.mapred.JobTracker: Adding task (JOB_CLEANUP) 'attempt_201307270122_0001_m_000003_0' to tip task_201307270122_0001_m_000003, for tracker 'tracker_slave1:localhost/127.0.0.1:58226'
2013-07-27 01:27:27,774 INFO org.apache.hadoop.mapred.JobInProgress: Task 'attempt_201307270122_0001_m_000003_0' has completed task_201307270122_0001_m_000003 successfully.
2013-07-27 01:27:27,775 INFO org.apache.hadoop.mapred.JobInProgress: Job job_201307270122_0001 has completed successfully.
2013-07-27 01:27:27,790 INFO org.apache.hadoop.mapred.JobInProgress$JobSummary: jobId=job_201307270122_0001,submitTime=1374913588466,launchTime=1374913589405,firstMapTaskLaunchTime=1374913599068,firstReduceTaskLaunchTime=1374913629494,firstJobSetupTaskLaunchTime=1374913589678,firstJobCleanupTaskLaunchTime=1374913641756,finishTime=1374913647775,numMaps=3,numSlotsPerMap=1,numReduces=1,numSlotsPerReduce=1,user=maedeh,queue=default,status=SUCCEEDED,mapSlotSeconds=67,reduceSlotsSeconds=10,clusterMapCapacity=4,clusterReduceCapacity=4
2013-07-27 01:27:28,328 INFO org.apache.hadoop.mapred.JobHistory: Creating DONE subfolder at file:/home/maedeh/hadoop-0.20.203.0/logs/history/done/version-1/master_1374913354885_/2013/07/27/000000
2013-07-27 01:27:28,330 INFO org.apache.hadoop.mapred.JobHistory: Moving file:/home/maedeh/hadoop-0.20.203.0/logs/history/job_201307270122_0001_1374913588466_maedeh_word+count to file:/home/maedeh/hadoop-0.20.203.0/logs/history/done/version-1/master_1374913354885_/2013/07/27/000000
2013-07-27 01:27:28,336 INFO org.apache.hadoop.mapred.JobHistory: Moving file:/home/maedeh/hadoop-0.20.203.0/logs/history/job_201307270122_0001_conf.xml to file:/home/maedeh/hadoop-0.20.203.0/logs/history/done/version-1/master_1374913354885_/2013/07/27/000000
2013-07-27 01:27:28,345 INFO org.apache.hadoop.mapred.JobTracker: Removing task 'attempt_201307270122_0001_m_000000_0'
2013-07-27 01:27:28,348 INFO org.apache.hadoop.mapred.JobTracker: Removing task 'attempt_201307270122_0001_m_000002_0'
2013-07-27 01:27:28,348 INFO org.apache.hadoop.mapred.JobTracker: Removing task 'attempt_201307270122_0001_m_000003_0'
2013-07-27 01:27:28,348 INFO org.apache.hadoop.mapred.JobTracker: Removing task 'attempt_201307270122_0001_m_000004_0'
2013-07-27 01:27:28,348 INFO org.apache.hadoop.mapred.JobTracker: Removing task 'attempt_201307270122_0001_r_000000_0'
2013-07-27 01:27:29,228 INFO org.apache.hadoop.mapred.JobTracker: Removing task 'attempt_201307270122_0001_m_000001_0'

字数示例:

maedeh@master:~/hadoop-0.20.203.0$ bin/hadoop jar hadoop*examples*.jar wordcount /maedeh/gutenberg /maedeh/gutenberg-output
13/07/27 01:26:27 INFO input.FileInputFormat: Total input paths to process : 3
13/07/27 01:26:28 INFO mapred.JobClient: Running job: job_201307270122_0001
13/07/27 01:26:29 INFO mapred.JobClient:  map 0% reduce 0%
13/07/27 01:27:01 INFO mapred.JobClient:  map 33% reduce 0%
13/07/27 01:27:03 INFO mapred.JobClient:  map 66% reduce 0%
13/07/27 01:27:07 INFO mapred.JobClient:  map 100% reduce 0%
13/07/27 01:27:22 INFO mapred.JobClient:  map 100% reduce 100%
13/07/27 01:27:28 INFO mapred.JobClient: Job complete: job_201307270122_0001
13/07/27 01:27:28 INFO mapred.JobClient: Counters: 25
13/07/27 01:27:28 INFO mapred.JobClient:   Job Counters 
13/07/27 01:27:28 INFO mapred.JobClient:     Launched reduce tasks=1
13/07/27 01:27:28 INFO mapred.JobClient:     SLOTS_MILLIS_MAPS=67684
13/07/27 01:27:28 INFO mapred.JobClient:     Total time spent by all reduces waiting after reserving slots (ms)=0
13/07/27 01:27:28 INFO mapred.JobClient:     Total time spent by all maps waiting after reserving slots (ms)=0
13/07/27 01:27:28 INFO mapred.JobClient:     Launched map tasks=3
13/07/27 01:27:28 INFO mapred.JobClient:     Data-local map tasks=3
13/07/27 01:27:28 INFO mapred.JobClient:     SLOTS_MILLIS_REDUCES=10249
13/07/27 01:27:28 INFO mapred.JobClient:   File Output Format Counters 
13/07/27 01:27:28 INFO mapred.JobClient:     Bytes Written=880838
13/07/27 01:27:28 INFO mapred.JobClient:   FileSystemCounters
13/07/27 01:27:28 INFO mapred.JobClient:     FILE_BYTES_READ=2214875
13/07/27 01:27:28 INFO mapred.JobClient:     HDFS_BYTES_READ=3671869
13/07/27 01:27:28 INFO mapred.JobClient:     FILE_BYTES_WRITTEN=3775263
13/07/27 01:27:28 INFO mapred.JobClient:     HDFS_BYTES_WRITTEN=880838
13/07/27 01:27:28 INFO mapred.JobClient:   File Input Format Counters 
13/07/27 01:27:28 INFO mapred.JobClient:     Bytes Read=3671523
13/07/27 01:27:28 INFO mapred.JobClient:   Map-Reduce Framework
13/07/27 01:27:28 INFO mapred.JobClient:     Reduce input groups=82335
13/07/27 01:27:28 INFO mapred.JobClient:     Map output materialized bytes=1474367
13/07/27 01:27:28 INFO mapred.JobClient:     Combine output records=102324
13/07/27 01:27:28 INFO mapred.JobClient:     Map input records=77931
13/07/27 01:27:28 INFO mapred.JobClient:     Reduce shuffle bytes=1474367
13/07/27 01:27:28 INFO mapred.JobClient:     Reduce output records=82335
13/07/27 01:27:28 INFO mapred.JobClient:     Spilled Records=255966
13/07/27 01:27:28 INFO mapred.JobClient:     Map output bytes=6076101
13/07/27 01:27:28 INFO mapred.JobClient:     Combine input records=629172
13/07/27 01:27:28 INFO mapred.JobClient:     Map output records=629172
13/07/27 01:27:28 INFO mapred.JobClient:     SPLIT_RAW_BYTES=346
13/07/27 01:27:28 INFO mapred.JobClient:     Reduce input records=102324
maedeh@master:~/hadoop-0.20.203.0$
z31licg0

z31licg01#

您可以进入jobtracker页面->作业历史->作业文件右侧链接->单击它->在哪里可以获得正在运行的计划程序信息来检查哪个计划程序正在运行。

相关问题