yarn+sqoop-after-ctrl+c=stuck-accepted

gk7wooem  于 2021-05-29  发布在  Hadoop
关注(0)|答案(1)|浏览(309)

我是hadoop新手,我正在用一个节点进行poc。这是第二次发生这种情况,不知道如何解决。它发生在使用ctrl+c停止map reduce之后。sqlserver正在通过sqoop将数据导入hbase,在出现错误并按ctrl+c组合键后,另一个作业开始工作,陷入接受状态。已排除所有作业应用程序。我重新启动了整个服务器。相应地配置了资源管理器(10gb内存,2个核心),但不知道会发生什么。

[root@hadoop01 /]# hadoop version
Hadoop 2.6.0.2.2.6.0-2800
Subversion git@github.com:hortonworks/hadoop.git -r acb70ecfae2c3c5ab46e24b0caebceaec16fdcd0
Compiled by jenkins on 2015-05-18T20:21Z
Compiled with protoc 2.5.0
From source with checksum a25c30f622eb057f47e2155f78dba5e
This command was run using /usr/hdp/2.2.6.0-2800/hadoop/hadoop-common-2.6.0.2.2.6.0-2800.jar

[root@hadoop01 fausto.branco]# cat /etc/centos-release
CentOS release 6.6 (Final)

sudo -u hdfs sqoop import –connect “jdbc:sqlserver://SQLServerIP:1433;database=db_POC;username=hdpteste;password=xxxxxx” \
–hbase-create-table \
–hbase-table vw_hdp_Arquivo \
–hbase-row-key “id_Arquivo, Valor” \
–column-family cf_name \
–driver com.microsoft.sqlserver.jdbc.SQLServerDriver \
–table “dbo.vw_hdp_Arquivo” -m 1

15/07/14 19:43:51 INFO zookeeper.ZooKeeper: Session: 0x14e8e1385000001 closed
15/07/14 19:43:51 INFO zookeeper.ClientCnxn: EventThread shut down
15/07/14 19:43:51 INFO impl.TimelineClientImpl: Timeline service address: hxxp://hadoop01.POC.local:8188/ws/v1/timeline/
15/07/14 19:43:51 INFO client.RMProxy: Connecting to ResourceManager at hadoop01.POC.local/XXX.XXX.XXX.XXX:8050
15/07/14 19:43:53 INFO db.DBInputFormat: Using read commited transaction isolation
15/07/14 19:43:53 INFO mapreduce.JobSubmitter: number of splits:1
15/07/14 19:43:54 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1436900603357_0002
15/07/14 19:43:54 INFO impl.YarnClientImpl: Submitted application application_1436900603357_0002
15/07/14 19:43:54 INFO mapreduce.Job: The url to track the job: hxxp://hadoop01.POC.local:8088/proxy/application_1436900603357_0002/
15/07/14 19:43:54 INFO mapreduce.Job: Running job: job_1436900603357_0002

[root@hadoop01 fausto.branco]# yarn application -status application_1436900603357_0002
15/07/14 19:45:05 INFO impl.TimelineClientImpl: Timeline service address: hxxp://hadoop01.POC.local:8188/ws/v1/timeline/
15/07/14 19:45:06 INFO client.RMProxy: Connecting to ResourceManager at hadoop01.POC.local/XXX.XXX.XXX.XXX:8050
Application Report :
Application-Id : application_1436900603357_0002
Application-Name : dbo.vw_hdp_Arquivo.jar
Application-Type : MAPREDUCE
User : hdfs
Queue : default
Start-Time : 1436903034500
Finish-Time : 0
Progress : 0%
State : ACCEPTED
Final-State : UNDEFINED
Tracking-URL : N/A
RPC Port : -1
AM Host : N/A
Aggregate Resource Allocation : 0 MB-seconds, 0 vcore-seconds
Diagnostics :

[root@hadoop01 fausto.branco]# hadoop job -status job_1436900603357_0002
DEPRECATED: Use of this script to execute mapred command is deprecated.
Instead use the mapred command for it.

15/07/14 19:46:19 INFO impl.TimelineClientImpl: Timeline service address: hxxp://hadoop01.POC.local:8188/ws/v1/timeline/
15/07/14 19:46:19 INFO client.RMProxy: Connecting to ResourceManager at hadoop01.POC.local/XXX.XXX.XXX.XXX:8050

Job: job_1436900603357_0002
Job File: /user/hdfs/.staging/job_1436900603357_0002/job.xml
Job Tracking URL : hxxp://hadoop01.POC.local:8088/proxy/application_1436900603357_0002/
Uber job : false
Number of maps: 0
Number of reduces: 0
map() completion: 0.0
reduce() completion: 0.0
Job state: PREP
retired: false
reason for failure:
Counters: 0
zzlelutf

zzlelutf1#

找到了!
分析此项目:聚合资源分配:0 mb秒,0 vcore秒
我看到节点状态不正常,在日志中发现此消息:
运行状况报告:1/1本地dir坏:/hadoop/yarn/local;1/1本地dir坏:/hadoop/yarn/log;
所以我放了更多的磁盘

相关问题