pig cassandra群集classnotfoundexception:org.apache.cassandra.hadoop.columnfamilysplit

zu0ti5jz  于 2021-06-21  发布在  Pig
关注(0)|答案(1)|浏览(325)

我正在尝试运行cassandra-0.8.5、hadoop 0.2.0和pig 0.8.1。我运行了一个非常简单的Pig脚本

rows = LOAD 'cassandra://pygmalion/$CF' USING CassandraStorage() AS (key, columns: bag {T: tuple(name, value)});
counted = foreach (group rows all) generate COUNT($1);
dump counted;

如果我运行本地模式,效果会很好。但如果我运行mapreduce模式,我总是会得到如下错误消息,我没办法了。任何帮助或帮助都将不胜感激。Pig_.log:

后端错误消息

java.io.IOException: java.lang.ClassNotFoundException: org.apache.cassandra.hadoop.ColumnFamilySplit
        at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigSplit.readFields(PigSplit.java:225)
        at org.apache.hadoop.io.serializer.WritableSerialization$WritableDeserializer.deserialize(WritableSerialization.java:67)
        at org.apache.hadoop.io.serializer.WritableSerialization$WritableDeserializer.deserialize(WritableSerialization.java:40)
        at org.apache.hadoop.mapred.MapTask.getSplitDetails(MapTask.java:349)
        at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:611)
        at org.apache.hadoop.mapred.MapTask.run(MapTask.java:323)
        at org.apache.hadoop.mapred.Child$4.run(Child.java:270)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:416)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1127)
        at org.apache.hadoop.mapred.Child.main(Child.java:264) Caused by: java.lang.ClassNotFoundException: org.apache.c

清管器堆放痕迹

ERROR 2997: Unable to recreate exception from backed error: java.io.IOException: java.lang.ClassNotFoundException: org.apache.cassandra.hadoop.ColumnFamilySplit

org.apache.pig.impl.logicalLayer.FrontendException: ERROR 1066: Unable to open iterator for alias counted. Backend error : Unable to recreate exception from backed error: java.io.IOException: java.lang.ClassNotFoundException: org.apache.cassandra.hadoop.ColumnFamilySplit
        at org.apache.pig.PigServer.openIterator(PigServer.java:753)
        at org.apache.pig.tools.grunt.GruntParser.processDump(GruntParser.java:615)
        at org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:303)
        at org.apache.pig.tools.grunt.GruntParser.loadScript(GruntParser.java:477)
        at org.apache.pig.tools.grunt.GruntParser.processScript(GruntParser.java:422)
        at org.apache.pig.tools.pigscript.parser.PigScriptParser.Script(PigScriptParser.java:692)
        at org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:425)
        at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:168)
        at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:144)
        at org.apache.pig.tools.grunt.Grunt.run(Grunt.java:76)
        at org.apache.pig.Main.run(Main.java:455)
        at org.apache.pig.Main.main(Main.java:107)
Caused by: org.apache.pig.backend.executionengine.ExecException: ERROR 2997: Unable to recreate exception from backed error: java.io.IOExcepti
on: java.lang.ClassNotFoundException: org.apache.cassandra.hadoop.ColumnFamilySplit
        at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.Launcher.getErrorMessages(Launcher.java:221)
        at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.Launcher.getStats(Launcher.java:151)
        at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher.launchPig(MapReduceLauncher.java:337)
        at org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.execute(HExecutionEngine.java:382)
        at org.apache.pig.PigServer.executeCompiledLogicalPlan(PigServer.java:1209)
        at org.apache.pig.PigServer.storeEx(PigServer.java:885)
        at org.apache.pig.PigServer.store(PigServer.java:827)
        at org.apache.pig.PigServer.openIterator(PigServer.java:739)
11dmarpk

11dmarpk1#

听起来您需要更新hadoop\u classpath环境变量以包含cassandra jar。完成后需要重新启动hadoop进程。

export HADOOP_CLASSPATH=/path/to/cassandra/lib/*:$HADOOP_CLASSPATH

看到了吗http://wiki.apache.org/cassandra/hadoopsupport 针对cassandra运行hadoop/pig时可能出现的其他问题。

相关问题