在spark java中,如何将文本文件转换为序列文件?以下是我的代码:
SparkConf sparkConf = new SparkConf().setAppName("txt2seq");
sparkConf.setMaster("local").set("spark.executor.memory", "1g");
sparkConf.set("spark.serializer", "org.apache.spark.serializer.KryoSerializer");
JavaSparkContext ctx = new JavaSparkContext(sparkConf);
JavaPairRDD<String, String> infile = ctx.wholeTextFiles("input_txt");
infile.saveAsNewAPIHadoopFile("outfile.seq", String.class, String.class, SequenceFileOutputFormat.class);
我得到了下面的错误。
14/12/07 23:43:33 ERROR Executor: Exception in task ID 0
java.io.IOException: Could not find a serializer for the Key class: 'java.lang.String'. Please ensure that the configuration 'io.serializations' is properly configured, if you're usingcustom serialization.
at org.apache.hadoop.io.SequenceFile$Writer.init(SequenceFile.java:1176)
at org.apache.hadoop.io.SequenceFile$Writer.<init>(SequenceFile.java:1091)
有人知道吗?谢谢您!
1条答案
按热度按时间kokeuurv1#
更改此项:
至