spark saveasnewapihadoopfile java.io.ioexception:找不到值类的序列化程序

mi7gmzs6 于 2021-05-29 发布在 Hadoop

关注(0)|答案(1)|浏览(398)

我尝试将java对rdd存储为hadoop序列文件，如下所示：

JavaPairRDD<ImmutableBytesWritable, Put> putRdd = ...
config.set("io.serializations","org.apache.hadoop.io.serializer.JavaSerialization,org.apache.hadoop.io.serializer.WritableSerialization");
putRdd.saveAsNewAPIHadoopFile(outputPath, ImmutableBytesWritable.class, Put.class, SequenceFileOutputFormat.class, config);

但即使我设定了 io.serializations :

2017-04-06 14:39:32,623 ERROR [Executor task launch worker-0] executor.Executor: Exception in task 0.0 in stage 0.0 (TID 0)
java.io.IOException: Could not find a serializer for the Value class: 'org.apache.hadoop.hbase.client.Put'. Please ensure that the configuration 'io.serializations' is properly configured, if you're usingcustom serialization.
    at org.apache.hadoop.io.SequenceFile$Writer.init(SequenceFile.java:1192)
    at org.apache.hadoop.io.SequenceFile$Writer.<init>(SequenceFile.java:1094)
    at org.apache.hadoop.io.SequenceFile.createWriter(SequenceFile.java:273)
    at org.apache.hadoop.io.SequenceFile.createWriter(SequenceFile.java:530)
    at org.apache.hadoop.mapreduce.lib.output.SequenceFileOutputFormat.getSequenceWriter(SequenceFileOutputFormat.java:64)
    at org.apache.hadoop.mapreduce.lib.output.SequenceFileOutputFormat.getRecordWriter(SequenceFileOutputFormat.java:75)
    at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsNewAPIHadoopDataset$1$$anonfun$12.apply(PairRDDFunctions.scala:1030)
    at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsNewAPIHadoopDataset$1$$anonfun$12.apply(PairRDDFunctions.scala:1014)
    at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:66)
    at org.apache.spark.scheduler.Task.run(Task.scala:88)
    at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:214)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)
2017-04-06 14:39:32,669 ERROR [task-result-getter-0] scheduler.TaskSetManager: Task 0 in stage 0.0 failed 1 times; aborting job

你知道我该怎么解决这个问题吗？？

Java hadoop hdfs apache-spark serialization

来源：https://stackoverflow.com/questions/43255698/spark-saveasnewapihadoopfile-java-io-ioexception-could-not-find-a-serializer-fo

1条答案

按热度按时间

o2g1uqev1#

我找到了解决办法，很明显，put（和所有hbase突变）有一个特定的序列化程序 MutationSerialization .
以下行修复了此问题：

config.setStrings("io.serializations", 
    config.get("io.serializations"),
    MutationSerialization.class.getName(),
    ResultSerialization.class.getName());

赞(0）回复(0）举报 2021-05-29

我来回答

spark saveasnewapihadoopfile java.io.ioexception:找不到值类的序列化程序

1条答案

相关问题

热门标签

最新问答