hadoopMap端连接条件任务

svmlkihl  于 2021-06-01  发布在  Hadoop
关注(0)|答案(1)|浏览(272)

我正在学习Hive图连接和遇到条件任务。我有以下关于条件任务的问题
什么是hadoop map join中的条件任务?
条件任务如何帮助识别map join中的小表?
以下Hive特性和它们的意义有什么区别
设置hive.auto.convert.join.noconditionaltask.size;
设置hive.mapjoin.smalltable.filesize;
有人能回答上面的问题并帮助我理解Map边连接中的条件任务吗。

btqmn9zl

btqmn9zl1#

1. What is a conditional task in hadoop map join ? 

     During compilation time, the query processor generates a
 conditional task containing a list of tasks and among this one of the tasks gets resolved to run during execution time. 

2. How does conditional task help in identifying the small table in map join? 

During the execution stage, the conditional task knows the exact file size of each input table, even if the table is an intermediate one. If all the tables are too large to be converted into map join, then just run the common join task as previously. If one of the tables is large and others are small enough to run map join, then the conditional task will pick the corresponding map join local task to run. By this mechanism, it can convert the common join into a map join automatically and dynamically. 

 3. What is difference between the following hive properties and their significance

    hive. Auto. Convert. Join

    default value: false

    This is used for auto join conversion. Once auto join is enabled, you need not specify the map-join in the query. 

 hive. Auto. Convert. Join. Noconditionaltask

 default value: true  

   This controls whether hive should enable the optimization of converting common join into map-join based on the input file size or not.If the total size of small tables is larger than 25MB, then the conditional task will choose the original common join to run.

 hive. Auto. Convert. Join. Noconditionaltask. Size

default value: 10000000

 If the sum of size for n-1 of the tables/partitions for an n-way join is smaller than this size, the join is directly converted to a map join (there is no conditional task). The default is 10mb. But this is dependent on hive. Auto. Convert. Join. Noconditionaltask and works only when it is true.

有关详细信息:https://cwiki.apache.org/confluence/download/attachments/27362054/hive+summit+2011-join.pdf

相关问题