使用cqlstorage存储来自pig的cql3Map

gzszwxb4  于 2021-06-21  发布在  Pig
关注(0)|答案(1)|浏览(360)

我已经用cql3在cassandra中创建了一个表,我想将pig中的一些数据存储到其中。我用

CREATE TABLE test
    (firstname VARCHAR, 
     surname VARCHAR, 
     averagekm DOUBLE, 
     firstyear INT, 
     lastyear INT, 
     km MAP<INT,INT>,
     PRIMARY KEY (firstname, surname)
    );

我在pig中的数据集是:

grunt> describe cqlFormat;
cqlFormat: {((chararray,Firstname: chararray), (chararray,Surname: chararray)), (Averagekm: double,FirstYear: int,LastYear: int,Km: (chararray, (Year: int,Km: int)))}

我想把它和

grunt> store cqlFormat into 'cql://test/test?output_query=UPDATE+test.test+set+averagekm+%3D+%3F%2C+firstyear+%3D+%3F%2C+lastyear+%3D+%3F%2C+km+%3D+%3F' using CqlStorage();

我将在日志文件中获取此错误:

Backend error message
---------------------
java.io.IOException: java.io.IOException: InvalidRequestException(why:Not enough bytes to read a map)
        at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigGenericMapReduce$Reduce.runPipeline(PigGenericMapReduce.java:465)
        at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigGenericMapReduce$Reduce.processOnePackageOutput(PigGenericMapReduce.java:428)
        at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigGenericMapReduce$Reduce.reduce(PigGenericMapReduce.java:408)
        at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigGenericMapReduce$Reduce.reduce(PigGenericMapReduce.java:262)
        at org.apache.hadoop.mapreduce.Reducer.run(Reducer.java:176)
        at org.apache.hadoop.mapred.ReduceTask.runNewReducer(ReduceTask.java:652)
        at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:420)
        at org.apache.hadoop.mapred.Child$4.run(Child.java:266)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:396)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1121)
        at org.apache.hadoop.mapred.Child.main(Child.java:260)
Caused by: java.io.IOException: InvalidRequestException(why:Not enough bytes to read a map)
        at org.apache.cassandra.hadoop.cql3.CqlRecordWriter$RangeClient.run(CqlRecordWriter.java:248)
Caused by: InvalidRequestException(why:Not enough bytes to read a map)
        at org.apache.cassandra.thrift.Cassandra$execute_prepared_cql3_query_result.read(Cassandra.java:41868)
        at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:78)
        at org.apache.cassandra.thrift.Cassandra$Client.recv_execute_prepared_cql3_query(Cassandra.java:1689)
        at org.apache.cassandra.thrift.Cassandra$Client.execute_prepared_cql3_query(Cassandra.java:1674)
        at org.apache.cassandra.hadoop.cql3.CqlRecordWriter$RangeClient.run(CqlRecordWriter.java:232)

数据的存储效果很好,因为存储时没有公里图。
我将datastax dse 3.1.2与pig 0.9.2(r1234438)、cqlsh 3.1.2、cassandra 1.2.6.5、cql spec 3.0.0一起使用。另外,我通过阅读链接创建了元组:
https://issues.apache.org/jira/browse/cassandra-5867?focusedcommentid=13753070&page=com.atlassian.jira.plugin.system.issuetabpanels:注解选项卡面板#注解-13753070
http://www.datastax.com/docs/datastax_enterprise3.1/solutions/about_pig#pig-读写
谢谢您!

dxpyg8gm

dxpyg8gm1#

dump cqlformat,显示数据格式。
确保它遵循(((firstname,somename),(姓氏,somename)),(averagekm,firstyear,lastyear,(map,(key1,value1),(key2,value2)))格式,然后它就可以工作了

相关问题