spark上的hive读取Parquet文件

smdncfj3 于 2021-05-29 发布在 Hadoop

关注(0)|答案(1)|浏览(476)

我正试着把Parquet文件读入spark的Hive里。
所以我发现我应该做点什么：

CREATE TABLE avro_test ROW FORMAT SERDE 'org.apache.hadoop.hive.serde2.avro.AvroSerDe' STORED 
AS AVRO TBLPROPERTIES ('avro.schema.url'='/files/events/avro_events_scheme.avsc'); 

CREATE EXTERNAL TABLE parquet_test LIKE avro_test STORED AS PARQUET LOCATION '/files/events/parquet_events/';

我的avro方案是：

{
 "type" : "parquet_file",
    "namespace" : "events",
    "name" : "events",
    "fields" : [
            { "name" : "category" , "type" : "string" },
            { "name" : "duration" , "type" : "long" },
            { "name" : "name" , "type" : "string" },
            { "name" : "user_id" , "type" : "string"},
            { "name" : "value" , "type" : "long" }
    ]
 }

结果我收到一个错误：

org.apache.spark.sql.catalyst.parser.ParseException: 
Operation not allowed: ROW FORMAT SERDE is incompatible with format 'avro', 
which also specifies a serde(line 1, pos 0)

hadoop Hive avro spark-avro parquet

来源：https://stackoverflow.com/questions/45241748/hive-on-spark-reading-parquet-file

1条答案

按热度按时间

50pmv0ei1#

I think we have to add inputforamt and outputformat classes. 

CREATE TABLE parquet_test
ROW FORMAT SERDE
   'org.apache.hadoop.hive.serde2.avro.AvroSerDe'
STORED AS INPUTFORMAT  
  'org.apache.hadoop.hive.ql.io.avro.AvroContainerInputFormat'
OUTPUTFORMAT
   'org.apache.hadoop.hive.ql.io.avro.AvroContainerOutputFormat'
TBLPROPERTIES (
  'avro.schema.url''avro.schema.url'='/hadoop/avro_events_scheme.avsc');

I hope above would work.

赞(0）回复(0）举报 2021-05-29

我来回答

spark上的hive读取Parquet文件

1条答案

相关问题

热门标签

最新问答