从远程hdfs读取文件

bvjveswy  于 2021-06-02  发布在  Hadoop
关注(0)|答案(1)|浏览(478)

我正在尝试从远程hdfs系统读取一个文件,并在本地计算机的控制台中显示。请注意,本地机器只能通过ssh密钥建立到任何hdfs节点的连接,ssh密钥的格式为.pem文件。
当我执行下面显示的代码时,程序运行,空闲一段时间,最后显示:

BlockMissingException : Could not obtain block

我的代码:

try {
            UserGroupInformation ugi = UserGroupInformation.createRemoteUser("remoteUser");

            ugi.doAs(new PrivilegedExceptionAction<Void>() {
                public Void run() throws Exception {
                    conf = new Configuration();
                    conf.set("fs.hdfs.impl", org.apache.hadoop.hdfs.DistributedFileSystem.class.getName());
                    conf.set("fs.file.impl", org.apache.hadoop.fs.LocalFileSystem.class.getName());
                    conf.addResource(new Path("/etc/hadoop/conf/core-site.xml"));
                    conf.addResource(new Path("/etc/hadoop/conf/hdfs-site.xml"));
                    conf.addResource(new Path("/etc/hadoop/conf/mapred-site.xml"));
                    conf.set("fs.default.name", hdfsurl);
                    conf.set("fs.defaultFS", hdfsurl);
                    conf.set("hadoop.job.ugi", "remoteUser");
                    conf.set("hadoop.ssl.enabled", "false");
                    readFromHDFS(hdfsurl);
                    return null;
                }
            });
        } catch (Exception e) {

public static void readFromHDFS(String hdfsURL) throws Exception {
        FileSystem fileSystem = FileSystem.get(conf);
        Path path = new Path(hdfsURL);
        if (!fileSystem.exists(path)) {
            System.out.println("File does not exists");
            return;
        }
        FSDataInputStream in = fileSystem.open(path);
        Scanner sc = new Scanner(in);
        while (sc.hasNextLine()) {
            System.out.println("line read from hdfs...." + sc.nextLine());
        }
        in.close();
        fileSystem.close();
}
iszxjhcz

iszxjhcz1#

1) 类型 hadoop fsck HDFS_FILE 检查特定的hdfs文件是否正常如果不正常,则说明特定的文件已损坏。删除损坏的文件并尝试下面的命令
2) 类型 hadoop dfsadmin -report 检查 Missing blocks: 0

相关问题