我正在尝试编写一个pig脚本,读取hbase表族,并将该族中找到的所有元数据写入elasticsearch。
让我展示一下我的table结构。首先,我将hbase表族作为Map[]加载:
hbasedata = LOAD 'hbase://tablename'
USING org.apache.pig.backend.hadoop.hbase.HBaseStorage('m:*', '-loadKey true')
AS (key:chararray, fields:map[]);
illustrate hbasedata;
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
| hbasedata | key:chararray | fields:map(:chararray) |
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
| | ea6d20a3-e978-4663-8c81-c92af65b8211 | {Field1=A, Field2=B, Field3=C} |
| | eb6d10a3-e978-2663-8a71-c92af65b8211 | {OtherField1=C, FieldZ=0} |
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
我需要这样在elasticsearch中编写元数据:
{
"_index" : "test",
"_type" : "test",
"_id" : "ea6d20a3-e978-4663-8c81-c92af65b8211",
"_score" : 1.0,
"_source" : {
"key" : "ea6d20a3-e978-4663-8c81-c92af65b8211",
"Field1" : "A",
"Field2" : "B",
"Field3" : "C"
},
{
"_index" : "test",
"_type" : "test",
"_id" : "eb6d10a3-e978-2663-8a71-c92af65b8211",
"_score" : 1.0,
"_source" : {
"key" : "eb6d10a3-e978-2663-8a71-c92af65b8211",
"OtherField1" : "C",
"FieldZ" : 0
}
我试过几种方法,但都没有成功。我开始认为,如果您不知道表的列,就不能得到这样的结果,那就很奇怪了,因为nosql的定义是无模式的。
有人能帮我摆脱吗?
暂无答案!
目前还没有任何答案,快来回答吧!