当我试图用maven(或者用ide intellij)编译pig udf时,出现以下错误:
cannot access org.apache.hadoop.io.WritableComparable
class file for org.apache.hadoop.io.WritableComparable not found
所以我想我应该在pom文件中添加一个对hadoop内核的依赖,但是仍然没有任何改变,尽管我检查了,writeablecomparable类在jar中。
我的自定义项类如下所示:
public class INCREMENTAL_UPDATE extends EvalFunc<DataBag> {
TupleFactory tupleFactory = TupleFactory.getInstance();
BagFactory bagFactory = BagFactory.getInstance();
public DataBag exec(Tuple input) throws IOException {
if (null == input || input.size() != 0) {
return null;
}
try {
DataBag inputbag = (DataBag) input.get(0);
Iterator it = inputbag.iterator();
DataBag outputbag = bagFactory.newDefaultBag();
Tuple previousTuple = null;
while (it.hasNext()) {
Tuple currentTuple = (Tuple) it.next();
Tuple outputTuple = tupleFactory.newTuple();
for (int i = 0; i < currentTuple.size(); i++) {
Object currentvalue = currentTuple.get(i);
if (currentvalue == null) {
outputTuple.append(currentvalue);
} else {
outputTuple.append(previousTuple.get(i));
}
}
previousTuple = outputTuple;
outputbag.add(outputTuple);
}
return outputbag;
} catch (Exception e) {
throw new IOException("UDF INCREMENTAL_UPDATE failed");
}
}
}
我还没有弄清楚这是否真的有效,因为我不确定是否分发了自定义项。如果是的话,以前的元组可能就不起作用了。但我甚至不能测试出来,因为我不能编译,因为上面的错误。
2条答案
按热度按时间ltqd579y1#
包括
hadoop-common-2.3.0.jar
应解决异常。5cnsuln72#
我也收到了这个错误,并最终能够通过包含hadoop类路径和pig.jar文件来编译。
如果这不起作用,请尝试不使用斜杠: