有没有人能提供一个示例代码段,说明如何将一个文件附加到现有的序列文件中?
下面是我用来附加到现有序列文件outputfile的代码,但在附加后读取序列文件时抛出校验和错误:
打开校验和文件时出现问题:/users/{homeditory}/desktop/sample/sequencefile/outputfile。忽略异常:java.io.eofexception
public class AppendSequenceFile {
/**
* @param args
* @throws IOException
* @throws IllegalAccessException
* @throws InstantiationException
*/
public static void main(String[] args) throws IOException,
InstantiationException, IllegalAccessException {
Configuration conf = new Configuration();
FileSystem fs = FileSystem.get(conf);
Path inputFile = new Path("/Users/{homedirectory}/Desktop/Sample/SequenceFile/sampleAppendTextFiles");
Path sequenceFile = new Path("/Users/{homedirectory}/Desktop/Sample/SequenceFile/outputfile");
FSDataInputStream inputStream;
Text key = new Text();
Text value = new Text();
SequenceFile.Writer writer = SequenceFile.createWriter(fs, conf,
sequenceFile, key.getClass(), value.getClass());
FileStatus[] fStatus = fs.listStatus(inputFile);
for (FileStatus fst : fStatus) {
String str = "";
System.out.println("Processing file : " + fst.getPath().getName() + " and the size is : " + fst.getPath().getName().length());
inputStream = fs.open(fst.getPath());
key.set(fst.getPath().getName());
while(inputStream.available()>0) {
str = str+inputStream.readLine();
}
value.set(str);
writer.append(key, value);
}
}
}
序列文件读取器:
public class SequenceFileReader{
public static void main(String[] args) throws Exception {
Configuration conf = new Configuration();
FileSystem fs = FileSystem.get(conf);
Path path = new Path("/Users/{homedirectory}/Desktop/Sample/SequenceFile/outputfile");
SequenceFile.Reader reader = null;
try {
reader = new SequenceFile.Reader(fs, path, conf);
Text key = new Text();
Text value = new Text();
while (reader.next(key, value)) { System.out.println(key);
System.out.println(value);
}
} finally {
IOUtils.closeStream(reader);
}
}
}
提前谢谢。
2条答案
按热度按时间a0x5cqrl1#
我自己并没有这样做,但是在浏览hadoopapi文档时我发现了这一点。
可以使用此api创建编写器。请参阅sequencefile
public static org.apache.hadoop.io.SequenceFile.Writer createWriter(FileContext fc,Configuration conf,Path name,Class keyClass,Class valClass,org.apache.hadoop.io.SequenceFile.CompressionType compressionType,CompressionCodec codec,org.apache.hadoop.io.SequenceFile.Metadata metadata,EnumSet<CreateFlag> createFlag,org.apache.hadoop.fs.Options.CreateOpts... opts) throws IOException
在这个api中,createflag可以帮助您指定“append”选项。c7rzv4ha2#
在使用sequencefile读取器对象从文件读取之前,请尝试关闭用于附加文件的sequencefile编写器对象。