after静态分区输出与hive中的预期不符

iqjalb3h 于 2021-05-29 发布在 Hadoop

关注(0)|答案(1)|浏览(293)

我正在使用静态分区数据进行如下处理

Id Name Salary Dept Doj
1,Murtaza,360000,Sales,2010
2,Soumya,478968,Admin,2011
3,Sneha,45789, Dev,2012
4,Asif ,145687, Qa,2012
5,Shreyashi,36598,Qa,2011
6,Adil,25987,Dev,2010
7,Yashwant,23982,Admin,2011
8,Mohsin,569875,2012
9,Anil,56798,Sales,2010
10,Balaji,56489,Sales,2012
11,Utsav,563895,Qa,2010
12,Anuj,546987,Dev,2010

用于创建分区表并将数据加载到其中的hql如下

create external table if not exists murtaza.PartSalaryReport (ID int,Name
string,Salary string,Dept string)
partitioned by (Doj string)
ROW FORMAT DELIMITED
FIELDS TERMINATED BY ','
stored as textfile
location '/user/cts573151/externaltables';

LOAD DATA  LOCAL INPATH '/home/cts573151/partition.txt'
overwrite into table murtaza.PartSalaryReport partition (Doj=2010);
LOAD DATA  LOCAL INPATH '/home/cts573151/partition.txt'
overwrite into table murtaza.PartSalaryReport partition (Doj=2011);
LOAD DATA  LOCAL INPATH '/home/cts573151/partition.txt'
overwrite into table murtaza.PartSalaryReport partition (Doj=2012);

Select * from murtaza.PartSalaryReport;`

现在的问题是，在外部表所在的hdfs位置，我应该获得数据目录，这样就可以了
`

[cts573151@aster2 ~]$ hadoop dfs -ls /user/cts573151/externaltables`
DEPRECATED: Use of this script to execute hdfs command is deprecated.
Instead use the hdfs command for it.

Found 4 items
drwxr-xr-x   - cts573151 supergroup          0 2016-12-12 13:06    /user/cts573151/externaltables/doj=2010
drwxr-xr-x   - cts573151 supergroup          0 2016-12-12 13:06  /user/cts573151/externaltables/doj=2011
drwxr-xr-x   - cts573151 supergroup          0 2016-12-12 13:06 /user/cts573151/externaltables/doj=2012

但是当我查看drwxr-xr-x-cts573151超级组0 2016-12-12 13:06/user/cts573151/externaltables/doj=2010中的数据时
它显示了所有20102011和2012年的数据，尽管它应该只显示2010年的数据

[cts573151@aster2 ~]$ hadoop dfs -ls /user/cts573151/externaltables/doj=2010
DEPRECATED: Use of this script to execute hdfs command is deprecated.
Instead use the hdfs command for it.

Found 1 items
-rwxr-xr-x   3 cts573151 supergroup        270 2016-12-12 13:06         /user/cts573151/externaltables/doj=2010/partition.txt
[cts573151@aster2 ~]$ hadoop dfs -cat   /user/cts573151/externaltables/doj=2010/partition.txt
DEPRECATED: Use of this script to execute hdfs command is deprecated.
Instead use the hdfs command for it.

1,Murtaza,360000,Sales,2010
2,Soumya,478968,Admin,2011
3,Sneha,45789,Dev,2012
4,Asif,145687,Qa,2012
5,Shreyashi,36598,Qa,2011
6,Adil,25987,Dev,2010
7,Yashwant,23982,Qa,2011
9,Anil,56798,Sales,2010
10,Balaji,56489,Sales,2012
11,Utsav,53895,Qa,2010
12,Anuj,54987,Dev,2010
[cts573151@aster2 ~]$

哪里不对？？？

hadoop Hive

来源：https://stackoverflow.com/questions/41096413/after-static-partitioning-output-is-not-as-expected-in-hive

1条答案

按热度按时间

bfrts1fy1#

由于要在配置单元中创建外部表，因此必须遵循以下命令集：

create external table if not exists murtaza.PartSalaryReport (
ID int, Name string, Salary string, Dept string)
partitioned by (Doj string)
ROW FORMAT DELIMITED
FIELDS TERMINATED BY ','
stored as textfile
location '/user/cts573151/externaltables';

alter table murtaza.PartSalaryReport add partition (Doj=2010);
hdfs dfs -put /home/cts573151/partition1.txt /user/cts573151/externaltables/Doj=2010/

alter table murtaza.PartSalaryReport add partition (Doj=2011);
hdfs dfs -put /home/cts573151/partition2.txt /user/cts573151/externaltables/Doj=2011/

alter table murtaza.PartSalaryReport add partition (Doj=2012);
hdfs dfs -put /home/cts573151/partition3.txt /user/cts573151/externaltables/Doj=2012/

这些命令对我有用，希望对你有帮助！！！

赞(0）回复(0）举报 2021-05-29

我来回答

after静态分区输出与hive中的预期不符

用于创建分区表并将数据加载到其中的hql如下

1条答案

相关问题

热门标签

最新问答