我试图读取mapreduce程序中的word文档文件,为此我使用了一个用户定义的 fileInputFormat
类作为 WordDocxInputFormat
以及 WordDocxInputFormatRecordReader
. 在 WordDocxInputFormatRecordReader
我用apachepoi来读单词 .docx
文件。但我得到了一个 java.lang.ClassNotFoundException
运行时错误。我正在Windows7平台上使用eclipse和hadoop-0.20.2。
我已经定义了我的 CLASSPATH
作为: JAVA_HOME\lib;C:\cygwin\home\bmohanty6\poijars\;
在 C:\cygwin\home\bmohanty6\poijars\
我保存了poi所需的jar文件(在附图中),并将它们添加到 Project->property->libraries->add external jar
.
我得到错误作为
13/09/17 12:35:26 INFO mapred.JobClient: Task Id : attempt_201309101108_0040_m_000000_2, Status : FAILED
Error: java.lang.ClassNotFoundException: org.apache.poi.xwpf.usermodel.XWPFDocument
at java.net.URLClassLoader$1.run(Unknown Source)
at java.net.URLClassLoader$1.run(Unknown Source)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(Unknown Source)
at java.lang.ClassLoader.loadClass(Unknown Source)
at sun.misc.Launcher$AppClassLoader.loadClass(Unknown Source)
at java.lang.ClassLoader.loadClass(Unknown Source)
at WordDocxInputFormat$WordDocxInputFormatRecordReader.next(WordDocxInputFormat.java:112)
at WordDocxInputFormat$WordDocxInputFormatRecordReader.next(WordDocxInputFormat.java:1)
at org.apache.hadoop.mapred.MapTask$TrackedRecordReader.moveToNext(MapTask.java:192)
at org.apache.hadoop.mapred.MapTask$TrackedRecordReader.next(MapTask.java:176)
at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:48)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:358)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:307)
at org.apache.hadoop.mapred.Child.main(Child.java:170)
这是我的worddocxinputformat.class
import java.io.IOException;
import java.util.Arrays;
import java.io.FileInputStream;
import java.util.logging.Level;
import java.util.logging.Logger;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.FSDataInputStream;
import org.apache.hadoop.fs.FileSystem;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.io.BytesWritable;
import org.apache.hadoop.io.DataOutputBuffer;
import org.apache.hadoop.io.NullWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapred.FileSplit;
import org.apache.hadoop.mapred.FileInputFormat;
import org.apache.hadoop.mapred.InputSplit;
import org.apache.hadoop.mapred.JobConf;
import org.apache.hadoop.mapred.RecordReader;
import org.apache.hadoop.mapred.Reporter;
import org.apache.hadoop.mapreduce.InputFormat;
import org.apache.hadoop.mapreduce.JobContext;
import org.apache.poi.xwpf.usermodel.*;
import org.apache.poi.xwpf.extractor.*;
/**
* Reads complete documents in Binary format.
*/
public class WordDocxInputFormat
extends FileInputFormat<Text, Text> {
public WordDocxInputFormat() {
super();
}
protected boolean isSplitable(FileSystem fs, Path filename) {
return false;
}
@Override
public RecordReader<Text, Text> getRecordReader(
InputSplit split, JobConf job, Reporter reporter) throws IOException {
return new WordDocxInputFormatRecordReader((FileSplit) split, job);
}
/**
* WordDocxInputFormatRecordReader class to read through a given binary document
* Outputs the filename along with the complete document
*/
public class WordDocxInputFormatRecordReader
implements RecordReader<Text, Text> {
private final FileSplit fileSplit;
private final Configuration conf;
private boolean processed = false;
public WordDocxInputFormatRecordReader(FileSplit fileSplit, Configuration conf)
throws IOException {
this.fileSplit = fileSplit;
this.conf = conf;
}
@Override
public Text createKey() {
return new Text();
}
@Override
public Text createValue() {
return new Text();
}
@Override
public long getPos() throws IOException {
return this.processed ? this.fileSplit.getLength() : 0;
}
@Override
public float getProgress() throws IOException {
return this.processed ? 1.0f : 0.0f;
}
@Override
public boolean next(Text key, Text value) throws IOException {
if (!this.processed) {
Path file = this.fileSplit.getPath();
try{
XWPFDocument docx = new XWPFDocument(new FileInputStream(file.toString()));
XWPFWordExtractor we = new XWPFWordExtractor(docx);
key.set(file.getName());
value.set(we.getText());
}
catch(IOException ex) {
Logger.getLogger(WordDocxInputFormatRecordReader.class.getName()).log(Level.SEVERE, null, ex);
}
this.processed = true;
return true;
}
else {
return false;
}
}
@Override
public void close() throws IOException {
}
}
}
1条答案
按热度按时间4urapxun1#
“但是我得到一个java.lang.classnotfoundexception运行时错误”
如果您在编译时遇到了这个错误和运行时错误,那么您几乎肯定在运行程序时使用了与编译时不同的设置。i、 e编译时,编译器会在有jar的区域中查找,但在运行程序时,它会在其他地方查找而没有找到它们,因此,错误只会在运行时出现。如果这确实是你的问题,我有几个建议:
如果您按照标签建议使用eclipse,请检查构建路径,我不知道eclipse如何编译和运行,因此您可能需要检查它。
或者,如果您将定期使用jar,您可以尝试将它们添加到jvm的外部lib中,如本教程所示,然后确保使用该jvm进行编译和运行。将它们放置在教程中显示的区域中,允许编译器在使用jvm时在编译和运行时自动检查jar。
希望这有帮助,祝你好运!