如何在hadoop上用opennlp训练意大利语模型?

rvpgvaaj  于 2021-06-02  发布在  Hadoop
关注(0)|答案(1)|浏览(410)

我想在hadoop上为意大利语实现一个自然语言处理算法
我有两个问题;
如何找到意大利语词干算法?
如何在hadoop中集成?
这是我的密码

String pathSent=...tagged sentences...;
String pathChunk=....chunked train path....;
File fileSent=new File(pathSent);
File fileChunk=new File(pathChunk);
InputStream inSent=null;
InputStream inChunk=null;

inSent = new FileInputStream(fileSent);
inChunk = new FileInputStream(fileChunk);
POSModel posModel=POSTaggerME.train("it", new WordTagSampleStream((
new InputStreamReader(inSent))), ModelType.MAXENT, null, null, 3, 3);

ObjectStream stringStream =new PlainTextByLineStream(new InputStreamReader(inChunk));
ObjectStream chunkStream = new ChunkSampleStream(stringStream);
ChunkerModel chunkModel=ChunkerME.train("it",chunkStream ,1, 1);
this.tagger= new POSTaggerME(posModel);
this.chunker=new ChunkerME(chunkModel);

inSent.close();
inChunk.close();
aor9mmx1

aor9mmx11#

你需要一个语法句子引擎:

"io voglio andare a casa"

io, sostantivo
volere, verbo
andare, verbo
a, preposizione semplice
casa, oggetto

当你标记了这个句子,你就可以教opennlp了。
在hadoop上创建自定义Map

public class Map extends Mapper<longwritable,
                            intwritable="" text,=""> {  

           private final static IntWritable one =
                           new IntWritable(1);  
          private Text word = new Text();    

          @Override  public void map(LongWritable key, Text value,
                      Context context)
      throws IOException, InterruptedException {

            //your code here
       } 
  }

在hadoop上创建自定义reduce

public class Reduce extends Reducer<text,
              intwritable,="" intwritable="" text,=""> {
 @Override
 protected void reduce(
   Text key,
   java.lang.Iterable<intwritable> values,
   org.apache.hadoop.mapreduce.Reducer<text,
           intwritable,="" intwritable="" text,="">.Context context)
   throws IOException, InterruptedException {
       // your reduce here
 }
}

配置两者

public static void main(String[] args)
                      throws Exception {
  Configuration conf = new Configuration();

  Job job = new Job(conf, "opennlp");
  job.setJarByClass(CustomOpenNLP.class);

  job.setOutputKeyClass(Text.class);
  job.setOutputValueClass(IntWritable.class);

  job.setMapperClass(Map.class);
  job.setReducerClass(Reduce.class);

  job.setInputFormatClass(TextInputFormat.class);
  job.setOutputFormatClass(TextOutputFormat.class);

  FileInputFormat.addInputPath(job, new Path(args[0]));
  FileOutputFormat.setOutputPath(job, new Path(args[1]));

  job.waitForCompletion(true);
}

相关问题