在hive中划分地理空间数据

rks48beu  于 2021-05-29  发布在  Hadoop
关注(0)|答案(0)|浏览(224)

我在为具有地理空间数据(即形状列)的表创建分区时遇到问题。我有一个包含地理空间数据的表构建,我想基于statename创建一个分区。
我创建了一个buildingpartiton表,如下所示

SET hive.exec.dynamic.partition=true;
 SET hive.exec.dynamic.partition.mode=non-strict;

 create table buildingpartition (OBJECTID string,
> RILUNIQUEID string,
> RILFEATURECODE string,
> BLDGNO string,
> BLDGNAME string,
> BLDGTYPE string,
> BLDGSUBTYPE string,
> BLDGCLASS string,
> BLDGROAD string,
> BLDGSUBROAD string,
> SUBLOCALITY string,
> BLDGSIZE string,
> TAG string,
> PINCODE string,
> NUMBEROFFLATS string,
> NUMBEROFSHOPS string,
> BLDG_TYPE string,
> CABLEOPERATORNAME string,
> AREA_1 string,
> LBU2 string,
> SHAPE binary)
> Partitioned by (STATENAME string)
> ROW FORMAT SERDE 'com.esri.hadoop.hive.serde.JsonSerde'
> STORED AS INPUTFORMAT 'com.esri.json.hadoop.UnenclosedJsonInputFormat'
>   OUTPUTFORMAT'org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat';

然后我尝试将数据从现有的构建表插入到这个表中。这个表格由4千万条记录组成。

> Insert into table buildingpartition parition (statename) select * from building;

我得到错误的主要原因是
引起原因:org.apache.hadoop.hive.ql.metadata.hiveexception:org.apache.hadoop.hive.ql.metadata.hiveexception:org.apache.hadoop.ipc.remoteexception(org.apache.hadoop.hdfs.protocol.fslimitexception$pathcomponenttoolongexception):statename的最大路径组件名称限制=00 00 00 03 00 00 83 54 0b b0 17 22 41 f0 6d 9e 04 7b 6143 41 c0 84 25 eb ed 17 22 41 a0 82 cd c2 80 61 43 41 01 00 00 00 05 00 00 00 00 00 00 83 54 0b b0 17 22 41 18 49 7a 80 61 43 41 40 a0 a2 83 ed 17 22 41 a0 82 cd c2 80 61 43 41 c0 84 25 eb ed 17 22 41 70 a7 13 4d 7b 61 43 41 40 c9 31 72 b0 17 22 41 f0 6d 9e 04 7b 61 43 41 00 83 54 0b b0 17 22 41 18 49 58 7a 80 61 43 41/apps/hive/warehouse/proddb.db/buildingpartition/.hive-staging\u hive\u 2016-12-15\u 10-58-15\u 294\u 7141760028369054723-1/\u task\u tmp.-ext-10002超出限制:长度=255 408
有人能帮我解决这个问题吗?我已附上下面的完整错误跟踪。任何帮助都将不胜感激。谢谢
错误:java.lang.runtimeexception:org.apache.hadoop.hive.ql.metadata.hiveexception:处理行{“objectid”:null,“riluniqueid”:“kol\u a\u lo0000\u 0597215”,“rilfeaturecode”:“lo0000”,“bldgno”:“null”,“bldgname”:“null”,“bldgtype”:“1”,“bldgsubtype”:“unclassified”,“bldgclass”:“building”,“bldgroad”:“purano bazar bridge”,“bldgsubroad”:“p k vishwas road”,“sublocality”:“milan garh colony”,“statename”:“west bengal”,“bldgsize”:“g+0”,“tag”:“null”,“pincode”:“700119”,“numberofflats”:“0”,“numberofshops”:“0”,“bldg\类型”:“others”,“cableoperatorname”:“null”,“area\ 1”:“0.0”,“lbu2”:“null”,“shape”:�t型�"一�米�{aca公司��%��"一����aca公司�t型�"艾克斯�aca公司@����"一����aca公司��%��"美联社�m{aca公司@�1r型�"一�米�{aca公司�t型�"艾克斯�aca}位于org.apache.hadoop.hive.ql.exec.mr.execmapper.map(execmapper。java:172)在org.apache.hadoop.mapred.maprunner.run(maprunner。java:54)在org.apache.hadoop.mapred.maptask.runoldmapper(maptask。java:453)在org.apache.hadoop.mapred.maptask.run(maptask。java:343)在org.apache.hadoop.mapred.yarnchild$2.run(yarnchild。java:168)在javax.security.auth.subject.doas(主题)中的java.security.accesscontroller.doprivileged(本机方法)。java:422)在org.apache.hadoop.security.usergroupinformation.doas(usergroupinformation。java:1709)在org.apache.hadoop.mapred.yarnchild.main(yarnchild。java:162)原因:org.apache.hadoop.hive.ql.metadata.hiveexception:配置单元运行时出错正在处理行{“objectid”:null,“riluniqueid”:“kol\u a\u lo0000\u 0597215”,“rilfeaturecode”:“lo0000”,“bldgno”:“null”,“bldgname”:“null”,“bldgtype”:“1”,“bldgsubtype”:“unclassified”,“bldgclass”:“building”,“bldgroad”:“purano bazar bridge”,“bldgsubrad”:“p k vishwas road”,“sublocality”:“milan garh colony”,“statename”:“west bengal”,“bldgsize”:“g+0”,“tag”:“null”,“pincode”:“700119”,“numberofflats”:“0”,“numberofshops”:“0”,“bldg_type”:“others”,“cableoperatorname”:“null”,“area_1”:“0.0”,“lbu2”:“null”,“shape”:�t型�"一�米�{aca公司��%��"一����aca公司�t型�"艾克斯�aca公司@����"一����aca公司��%��"美联社�m{aca公司@�1r型�"一�米�{aca公司�t型�"艾克斯�aca}位于org.apache.hadoop.hive.ql.exec.mapoperator.process(mapoperator。java:562)在org.apache.hadoop.hive.ql.exec.mr.execmapper.map(execmapper。java:163) ... 8更多原因:org.apache.hadoop.hive.ql.metadata.hiveexception:org.apache.hadoop.hive.ql.metadata.hiveexception:org.apache.hadoop.ipc.remoteexception(org.apache.hadoop.hdfs.protocol.fslimitexception$pathcomponenttoolongexception):statename的最大路径组件名称限制=00 00 00 03 00 00 00 00 00 83 54 0b b0 17 22 41 f0 6d 9e04 7b 61 43 41 c0 84 25 eb ed 17 22 41 a0 82 cd c2 80 61 43 41 01 00 00 00 05 00 00 00 00 00 83 54 0b b0 17 22 41 18 49 58 7a 80 61 43 41 40 a0 a2 83 ed 17 22 41 a0 82 cd c2 80 61 43 41 c0 84 25 eb ed 17 22 41 70 a7 13 4d 7b 61 43 41 40 c9 31 72 b0 17 22 41 f0 6d 9e 04 7b 61 43 41 00 54 0b b0 17 22 41 18 49 58 7a 80 61 43 41directory/apps/hive/warehouse/proddb.db/buildingpartition/.hive-staging\u hive\u 2016-12-15\u 10-58-15\u 294\u 7141760028369054723-1/\u task\u tmp.-ext-10002超出限制:limit=255 length=408 at org.apache.hadoop.hdfs.server.namenode.fsdirectory.verifymaxcomponentlength(fsdirectory)。java:949)在org.apache.hadoop.hdfs.server.namenode.fsdirectory.addlastinode(fsdirectory。java:1014)在org.apache.hadoop.hdfs.server.namenode.fsdirmkdirop.unprotectedmkdir(fsdirmkdirop。java:237)在org.apache.hadoop.hdfs.server.namenode.fsdirmkdirop.createsingledirectory(fsdirmkdirop。java:191)在org.apache.hadoop.hdfs.server.namenode.fsdirmkdirop.createchildrendirectories(fsdirmkdirop。java:166)在org.apache.hadoop.hdfs.server.namenode.fsdirmkdirop.createancestordirectories(fsdirmkdirop。java:134)位于org.apache.hadoop.hdfs.server.namenode.fsnamesystem.startfileinternal(fsnamesystem)。java:2615)在org.apache.hadoop.hdfs.server.namenode.fsnamesystem.startfileint(fsnamesystem)。java:2493)位于org.apache.hadoop.hdfs.server.namenode.fsnamesystem.startfile(fsnamesystem)。java:2377)在org.apache.hadoop.hdfs.server.namenode.namenoderpcserver.create(namenoderpcserver。java:708)在org.apache.hadoop.hdfs.protocolpb.clientnamenodeprotocolserversidetranslatorpb.create(clientnamenodeprotocolserversidetranslatorpb。java:405)org.apache.hadoop.hdfs.protocol.proto.clientnamenodeprotocolprotos$clientnamenodeprotocol$2.callblockingmethod(clientnamenodeprotocolprotos.java)位于org.apache.hadoop.ipc.protobufrpceengine$server$protobufrpinvoker.call(protobufrpceengine。java:616)在org.apache.hadoop.ipc.rpc$server.call(rpc。java:969)在org.apache.hadoop.ipc.server$handler$1.run(server。java:2206)在org.apache.hadoop.ipc.server$handler$1.run(server。java:2202)位于java.security.accesscontroller.doprivileged(本机方法)javax.security.auth.subject.doas(主题。java:422)在org.apache.hadoop.security.usergroupinformation.doas(usergroupinformation。java:1709)在org.apache.hadoop.ipc.server$handler.run(server。java:2200)在org.apache.hadoop.hive.ql.exec.filesinkoperator.createbucketfiles(filesinkoperator。java:577)在org.apache.hadoop.hive.ql.exec.filesinkoperator.createnewpaths(filesinkoperator)。java:859)位于org.apache.hadoop.hive.ql.exec.filesinkoperator.getdynoutpaths(filesinkoperator)。java:966)在org.apache.hadoop.hive.ql.exec.filesinkoperator.process(filesinkoperator。java:709)在org.apache.hadoop.hive.ql.exec.operator.forward(operator。java:838)在org.apache.hadoop.hive.ql.exec.selectoperator.process(selectoperator。java:88)在org.apache.hadoop.hive.ql.exec.operator.forward(operator。java:838)在org.apache.hadoop.hive.ql.exec.tablescanoperator.process(tablescanoperator。java:117)在org.apache.hadoop.hive.ql.exec.mapoperator$mapopctx.forward(mapoperator。java:167)在org.apache.hadoop.hive.ql.exec.mapoperator.process(mapoperator。java:552) ... 9更多原因:org.apache.hadoop.hive.ql.metadata.hiveexception:org.apache.hadoop.ipc.remoteexception(org.apache.hadoop.hdfs.protocol.fslimitexception$pathcomponenttoolongexception):statename的最大路径组件名称限制=00 00 00 03 05 00 00 00 00 00 83 54 0b b0 17 22 41 f0 6d 9e 04 7b 61 43 41 c0 84 25 eb ed 17 22 41 a0 82 cd c280 61 43 41 01 00 00 00 05 00 00 00 00 00 00 00 00 83 54 0b b0 17 22 41 18 49 58 7a 80 61 43 41 40 a0 a2 83 ed 17 22 41 a0 82 cd c2 80 61 43 41 c0 84 25 eb ed 17 22 41 70 a7 13 4d 7b 61 43 41 40 c9 31 72 b0 17 22 41 f0 6d 9e 04 7b 61 43 41 00 83 54 0b b0 17 22 41 18 49 58 7a 80 61 43 41/apps/hive/warehouse/proddb.db/buildingpartition/.hive-staging\u hive\u 2016-12-15\u 10-58-15\u 294\u 7141760028369054723-1/\u task\u tmp.-ext-10002超出限制:limit=255 length=408 at org.apache.hadoop.hdfs.server.namenode.fsdirectory.verifymaxcomponentlength(fsdirectory)。java:949)在org.apache.hadoop.hdfs.server.namenode.fsdirectory.addlastinode(fsdirectory。java:1014)在org.apache.hadoop.hdfs.server.namenode.fsdirmkdirop.unprotectedmkdir(fsdirmkdirop。java:237)在org.apache.hadoop.hdfs.server.namenode.fsdirmkdirop.createsingledirectory(fsdirmkdirop。java:191)在org.apache.hadoop.hdfs.server.namenode.fsdirmkdirop.createchildrendirectories(fsdirmkdirop。java:166)在org.apache.hadoop.hdfs.server.namenode.fsdirmkdirop.createancestordirectories(fsdirmkdirop。java:134)位于org.apache.hadoop.hdfs.server.namenode.fsnamesystem.startfileinternal(fsnamesystem)。java:2615)在org.apache.hadoop.hdfs.server.namenode.fsnamesystem.startfileint(fsnamesystem)。java:2493)位于org.apache.hadoop.hdfs.server.namenode.fsnamesystem.startfile(fsnamesystem)。java:2377)在org.apache.hadoop.hdfs.server.namenode.namenoderpcserver.create(namenoderpcserver。java:708)在org.apache.hadoop.hdfs.protocolpb.clientnamenodeprotocolserversidetranslatorpb.create(clientnamenodeprotocolserversidetranslatorpb。java:405)org.apache.hadoop.hdfs.protocol.proto.clientnamenodeprotocolprotos$clientnamenodeprotocol$2.callblockingmethod(clientnamenodeprotocolprotos.java)位于org.apache.hadoop.ipc.protobufrpceengine$server$protobufrpinvoker.call(protobufrpceengine。java:616)在org.apache.hadoop.ipc.rpc$server.call(rpc。java:969)在org.apache.hadoop.ipc.server$handler$1.run(server。java:2206)在org.apache.hadoop.ipc.server$handler$1.run(server。java:2202)位于java.security.accesscontroller.doprivileged(本机方法)javax.security.auth.subject.doas(主题。java:422)在org.apache.hadoop.security.usergroupinformation.doas(usergroupinformation。java:1709)在org.apache.hadoop.ipc.server$handler.run(server。java:2200)在org.apache.hadoop.hive.ql.io.hivefileformatutils.gethiverecordwriter(hivefileformatutils。java:249)在org.apache.hadoop.hive.ql.exec.filesinkoperator.createbucketforfileidx(filesinkoperator)。java:622)在org.apache.hadoop.hive.ql.exec.filesinkoperator.createbucketfiles(filesinkoperator。java:566) ... 还有18个原因:org.apache.had

暂无答案!

目前还没有任何答案,快来回答吧!

相关问题