我的spark程序从elasticsearch读取数据(以javapairdd>的形式)。这个javapairdd使用以下内部类转换为javardd
static class Transformer implements
Function<Tuple2<String, Map<String, Object>>, LabeledPoint> {
@Override
public LabeledPoint call(Tuple2<String, Map<String, Object>> arg0)
throws Exception {
HashingTF tf = new HashingTF();
Map<String, Object> map = (Map<String, Object>) arg0._2();
// get values from Map
Set<String> keys = map.keySet();
List<Object> valuesList = new ArrayList<Object>();
for (Iterator<String> i = keys.iterator(); i.hasNext();) {
String key = (String) i.next();
Object value = (Object) map.get(key);
valuesList.add(value);
}
return new LabeledPoint(1d, tf.transform(valuesList));
}
}
使用javardd数据生成随机森林模型并保存到同一台机器上的特定位置。模型保存成功。
但在加载模型时,我遇到了以下异常,
sun.reflect.nativemethodaccessorimpl.invoke0(本机方法)处的java.lang.reflect.invokeTargetException at sun.reflect.nativemethodaccessorimpl.invoke(本机方法)。java:62)在sun.reflect.delegatingmethodaccessorimpl.invoke(delegatingmethodaccessorimpl。java:43)在java.lang.reflect.method.invoke(方法。java:497)在org.codehaus.mojo.exec.execjavamojo$1.run(execjavamojo。java:293)在java.lang.thread.run(线程。java:745)原因:org.json4s.package$mappingexception:在org.json4s.reflect.package$.fail(package)中未找到可转换为java.lang.string的值。scala:96)在org.json4s.extraction$.convert(extraction。scala:554)在org.json4s.extraction$.extract(提取。scala:331)在org.json4s.extraction$.extract(extraction。scala:42)在org.json4s.extractablejsonastnode.extract(extractablejsonastnode。scala:21)在org.apache.spark.mllib.tree.model.decisiontreemodel$.load(decisiontreemodel。scala:326)在org.apache.spark.mllib.tree.model.decisiontreemodel.load(decisiontreemodel.scala)位于co.nttd.integration.sparkesintegration.main(sparkesintegration)。java:96)
为了保存模型,编写了以下代码
JavaPairRDD<String, Map<String, Object>> esRDD = JavaEsSpark
.esRDD(javaSparkContext);
// Transform data in JavaRDD<LabeledPoint>
JavaRDD<LabeledPoint> data = esRDD.map(new Transformer());
final RandomForestModel model = RandomForest.trainClassifier(
inputTrainingData, numClasses, categoricalFeaturesInfo, numTrees,
featureSubsetStrategy, impurity, maxDepth, maxBins, seed);
model.save(JavaSparkContext.toSparkContext(jsc),
"myPath");
要加载模型,decisiontreemodel model=decisiontreemodel.load(sc.sc(),“file:///mypath”);
暂无答案!
目前还没有任何答案,快来回答吧!