我正试图在cloudera的Hive中创建一个带扣的table。但是,创建的普通表没有任何存储桶。
首先,我使用hivecli创建了一个普通表,它使用了named marks\u temp
CREATE TABLE marks_temp(
id INT,
Name string,
mark int
)
ROW FORMAT DELIMITED
FIELDS TERMINATED BY ',';
我已经从一个文本文件“desktop/data/littlebigdata.txt”将以下数据加载到marks\ temp表中
101,Firdaus,88
102,Pranav,78
103,Rahul,65
104,Sanjoy,65
105,Firdaus,88
106,Pranav,78
107,Rahul,65
108,Sanjoy,65
109,Amar,54
110,Sahil,34
111,Rahul,45
112,Rajnish,67
113,Ranjeet,56
114,Sanjoy,34
我已经用下面的命令加载了上面的数据
LOAD DATA LOCAL INPATH 'Desktop/Data/littlebigdata.txt'
INTO TABLE marks_temp;
在成功加载数据之后,我将创建一个名为marks\u temp的bucked表
CREATE TABLE marks_bucketed(
id INT,
Name string,
mark int
)
CLUSTERED BY (id) INTO 4 BUCKETS;
现在,我将数据从marks\u temp表插入marks\u bucketed表。
INSERT INTO marks_bucketed
SELECT id,Name, mark FROM marks_temp;
在此之后,一些作业开始运行。什么,我在作业日志中看到它说“由于没有reduce操作符,reduce任务的数量被设置为0”
hive> insert into marks_bucketed
> select id,Name,mark from marks_temp;
Query ID = cloudera_20180601035353_29b25ffe-541e-491e-aea6-b36ede88ed79
Total jobs = 3
Launching Job 1 out of 3
Number of reduce tasks is set to 0 since there's no reduce operator
Starting Job = job_1527668582032_0004, Tracking URL = http://quickstart.cloudera:8088/proxy/application_1527668582032_0004/
Kill Command = /usr/lib/hadoop/bin/hadoop job -kill job_1527668582032_0004
Hadoop job information for Stage-1: number of mappers: 1; number of reducers: 0
2018-06-01 03:54:01,328 Stage-1 map = 0%, reduce = 0%
2018-06-01 03:54:14,444 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 2.21 sec
MapReduce Total cumulative CPU time: 2 seconds 210 msec
Ended Job = job_1527668582032_0004
Stage-4 is selected by condition resolver.
Stage-3 is filtered out by condition resolver.
Stage-5 is filtered out by condition resolver.
Moving data to: hdfs://quickstart.cloudera:8020/user/hive/warehouse/marks_bucketed/.hive-staging_hive_2018-06-01_03-53-45_726_2788383119636056364-1/-ext-10000
Loading data to table default.marks_bucketed
Table default.marks_bucketed stats: [numFiles=1, numRows=14, totalSize=194, rawDataSize=180]
MapReduce Jobs Launched:
Stage-Stage-1: Map: 1 Cumulative CPU: 2.21 sec HDFS Read: 3937 HDFS Write: 273 SUCCESS
Total MapReduce CPU Time Spent: 2 seconds 210 msec
OK
Time taken: 31.307 seconds
甚至,hue文件浏览器也只显示一个文件。附上截图。色相文件浏览器屏幕截图为马克琰扣表
1条答案
按热度按时间nkcskrwz1#
从配置单元文档
仅版本0.x和1.x
命令set hive.enforce.bucketing=true;允许根据表自动选择正确的还原数和按列分类。否则,您需要将reducer的数量设置为与set mapred.reduce.tasks=256中的bucket数量相同;有一个群集。。。select中的子句。
所以您需要设置属性来强制bucketing,或者选择manual选项并像