使用pig或hive将csv数据加载到hbase

pkmbmrz7 于 2021-06-03 发布在 Hadoop

关注(0)|答案(3)|浏览(454)

嗨，我创建了一个pig脚本，将数据加载到hbase中。我的csv文件存储在hadoop的 /hbase_tables/zip.csv Pig手稿

register /home/hduser/pig-0.12.0/lib/pig-0.8.0-core.jar;
A = LOAD '/hbase_tables/zip.csv' USING PigStorage(',') as (id:chararray, zip:chararray, desc1:chararray, desc2:chararray, income:chararray);
STORE A INTO 'hbase://mydata' USING org.apache.pig.backend.hadoop.hbase.HBaseStorage('zip:zip,desc:desc1,desc:desc2,income:income');

当我执行时，它给出以下错误

清管器堆放痕迹

ERROR 2017: Internal error creating job configuration.

org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobCreationException: ERROR 2017: Internal error creating job configuration.
        at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler.getJob(JobControlCompiler.java:667)
        at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler.compile(JobControlCompiler.java:256)
        at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher.launchPig(MapReduceLauncher.java:147)
        at org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.execute(HExecutionEngine.java:378)
        at org.apache.pig.PigServer.executeCompiledLogicalPlan(PigServer.java:1198)
        at org.apache.pig.PigServer.execute(PigServer.java:1190)
        at org.apache.pig.PigServer.access$100(PigServer.java:128)
        at org.apache.pig.PigServer$Graph.execute(PigServer.java:1517)
        at org.apache.pig.PigServer.executeBatchEx(PigServer.java:362)
        at org.apache.pig.PigServer.executeBatch(PigServer.java:329)
        at org.apache.pig.tools.grunt.GruntParser.executeBatch(GruntParser.java:112)
        at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:169)
        at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:141)
        at org.apache.pig.tools.grunt.Grunt.exec(Grunt.java:90)
        at org.apache.pig.Main.run(Main.java:510)
        at org.apache.pig.Main.main(Main.java:107)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:606)
        at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
Caused by: java.lang.IllegalArgumentException: java.net.URISyntaxException: Relative path in absolute URI: hbase://mydata_logs
        at org.apache.hadoop.fs.Path.initialize(Path.java:148)
        at org.apache.hadoop.fs.Path.<init>(Path.java:71)
        at org.apache.hadoop.fs.Path.<init>(Path.java:45)
        at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler.getJob(JobControlCompiler.java:470)
        ... 20 more
Caused by: java.net.URISyntaxException: Relative path in absolute URI: hbase://mydata_logs
        at java.net.URI.checkPath(URI.java:1804)
        at java.net.URI.<init>(URI.java:752)
        at org.apache.hadoop.fs.Path.initialize(Path.java:145)
        ... 23 more

请告诉我如何将csv数据文件导入hbase，或者您是否有其他解决方案。

hadoop Hive hbase apache-pig hbasestorage

来源：https://stackoverflow.com/questions/23366249/load-csv-data-to-hbase-using-pig-or-hive

3条答案

按热度按时间

50pmv0ei1#

我曾经有过同样的任务，并且有完全可行的解决方案（实际上，我不确定您的第三行代码中是否有逗号）：

%default hbase_home `echo \$HBASE_HOME`;
%default tmp '/user/alexander/tmp/users_dump/k14'

set zookeeper.znode.parent '/hbase-unsecure';
set hbase.zookeeper.quorum 'dmp-hbase.local';

register $hbase_home/lib/zookeeper-3.4.5.jar;
register $hbase_home/hbase-0.94.20.jar;

UsersHdfs = LOAD '$tmp' using PigStorage('\t', '-schema');
store UsersHdfs into 'hbase://user_test' using
    org.apache.pig.backend.hadoop.hbase.HBaseStorage(
        'id:DEFAULT id:last_modified birth:year gender:female gender:male','-caster HBaseBinaryConverter'
);

那代码对我有用，也许问题出在你的hbase配置上。你可以提供你的.csv文件，我们可以更详细地讨论它。

赞(0）回复(0）举报 2021-06-03

ncecgwcz2#

似乎您的问题是绝对uri中的“相对路径”：hbase://mydata_logs. 你确定路径正确吗？

赞(0）回复(0）举报 2021-06-03

eeq64g8w3#

可能是table mydata_logs 不存在。开始时间： hbase shell 和类型 list . 这是你的table吗 mydata_logs 在名单上？

赞(0）回复(0）举报 2021-06-03