我正在尝试从远程hdfs系统读取一个文件,并在本地计算机的控制台中显示。请注意,本地机器只能通过ssh密钥建立到任何hdfs节点的连接,ssh密钥的格式为.pem文件。
当我执行下面显示的代码时,程序运行,空闲一段时间,最后显示:
BlockMissingException : Could not obtain block
我的代码:
try {
UserGroupInformation ugi = UserGroupInformation.createRemoteUser("remoteUser");
ugi.doAs(new PrivilegedExceptionAction<Void>() {
public Void run() throws Exception {
conf = new Configuration();
conf.set("fs.hdfs.impl", org.apache.hadoop.hdfs.DistributedFileSystem.class.getName());
conf.set("fs.file.impl", org.apache.hadoop.fs.LocalFileSystem.class.getName());
conf.addResource(new Path("/etc/hadoop/conf/core-site.xml"));
conf.addResource(new Path("/etc/hadoop/conf/hdfs-site.xml"));
conf.addResource(new Path("/etc/hadoop/conf/mapred-site.xml"));
conf.set("fs.default.name", hdfsurl);
conf.set("fs.defaultFS", hdfsurl);
conf.set("hadoop.job.ugi", "remoteUser");
conf.set("hadoop.ssl.enabled", "false");
readFromHDFS(hdfsurl);
return null;
}
});
} catch (Exception e) {
public static void readFromHDFS(String hdfsURL) throws Exception {
FileSystem fileSystem = FileSystem.get(conf);
Path path = new Path(hdfsURL);
if (!fileSystem.exists(path)) {
System.out.println("File does not exists");
return;
}
FSDataInputStream in = fileSystem.open(path);
Scanner sc = new Scanner(in);
while (sc.hasNextLine()) {
System.out.println("line read from hdfs...." + sc.nextLine());
}
in.close();
fileSystem.close();
}
1条答案
按热度按时间iszxjhcz1#
1) 类型
hadoop fsck HDFS_FILE
检查特定的hdfs文件是否正常如果不正常,则说明特定的文件已损坏。删除损坏的文件并尝试下面的命令2) 类型
hadoop dfsadmin -report
检查Missing blocks: 0