hadoop—是否可以在HDF上运行配置单元查询而不使用yarn?

e5nqia27  于 2021-05-27  发布在  Hadoop
关注(0)|答案(0)|浏览(191)

我已经使用hive和hadoop在伪分布式模式下配置了yarn,并且我的查询正确运行并给出了预期的结果。现在对于我的项目,我必须在伪分布式模式下运行相同的配置单元查询,而不使用yarn和本地模式。
要在伪分布式模式下运行hadoop而不使用yarn,我只删除了文件中的属性:etc/hadoop/mapred-site.xml和etc/hadoop/yarn-site.xml,但是当我要在hdfs中创建表时,总是会遇到同样的问题,他似乎找不到任何缩减器:

Total jobs = 3
Launching Job 1 out of 3
Number of reduce tasks is set to 0 since there's no reduce operator
Job running in-process (local Hadoop)
2020-12-17 18:35:02,251 Stage-1 map = 0%,  reduce = 0%
2020-12-17 18:35:14,304 Stage-1 map = 100%,  reduce = 0%
Ended Job = job_local1317984459_0001
Stage-4 is selected by condition resolver.
Stage-3 is filtered out by condition resolver.
Stage-5 is filtered out by condition resolver.
  [......]
 Time taken to load dynamic partitions: 16.956 seconds
     Time taken for adding to write entity : 0.016 seconds
MapReduce Jobs Launched: 
Stage-Stage-1:  HDFS Read: 2201663 HDFS Write: 2327026 SUCCESS
Total MapReduce CPU Time Spent: 0 msec
OK
Time taken: 56.534 seconds

相反,当我要运行查询时,出现以下错误:

Total jobs = 2
Launching Job 1 out of 2
Number of reduce tasks not specified. Defaulting to jobconf value of: 1
In order to change the average load for a reducer (in bytes):
  set hive.exec.reducers.bytes.per.reducer=<number>
In order to limit the maximum number of reducers:
  set hive.exec.reducers.max=<number>
In order to set a constant number of reducers:
  set mapreduce.job.reduces=<number>
Job running in-process (local Hadoop)
2020-12-17 19:39:34,746 Stage-1 map = 0%,  reduce = 0%
Ended Job = job_local1837749065_0002 with errors
Error during job, obtaining debugging information...
Job Tracking URL: http://localhost:8080/
FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.mr.MapRedTask
MapReduce Jobs Launched: 
Stage-Stage-1:  HDFS Read: 0 HDFS Write: 0 FAIL
Total MapReduce CPU Time Spent: 0 msec

如何在没有Yarn的伪分布式和本地模式下运行配置单元查询?

暂无答案!

目前还没有任何答案,快来回答吧!

相关问题