使用JavaAPI从hdfs(hortonworks沙盒)读取文件时出现异常

v440hwme  于 2021-06-02  发布在  Hadoop
关注(0)|答案(1)|浏览(428)

我在尝试使用javaapi从hdfs(hortonworkssandbox)读取文件时遇到了问题。下面是我的代码-

System.setProperty ("hadoop.home.dir", "/");
    URI uri = URI.create ("hdfs://localhost:8020/user/maria_dev/test.txt");

    Path path = new Path (uri);

    Configuration conf = new Configuration ();
    conf.set ("fs.defaultFS", "hdfs://localhost:8020");
    conf.set ("dfs.client.use.datanode.hostname","true");
    conf.set("dfs.datanode.use.datanode.hostname","true");
    conf.set("dfs.client.use.legacy.blockreader", "true");

    byte[] btbuffer = new byte[5];
    String s;
    try (FileSystem fs = FileSystem.get (uri, conf)) {
        try {
            FSDataInputStream fileIn = fs.open (path);
            //s = fileIn.readUTF ();
            fileIn.read (btbuffer, 0, 20); 
            s = new String (btbuffer, Charset.forName ("UTF-8"));

        } catch (Exception e) {
            e.printStackTrace ();
        }
    }
    catch (Exception err){
        err.printStackTrace ();
    }

以下是我得到的例外-

10:39:51.803 [main] WARN org.apache.hadoop.hdfs.BlockReaderFactory - I/O error constructing remote block reader. java.net.ConnectException: Connection refused .   
10:39:51.803 [main] WARN org.apache.hadoop.hdfs.DFSClient - Failed to connect to sandbox.hortonworks.com/172.17.0.2:50010 for block, add to deadNodes and continue. java.net.ConnectException: Connection refused .   
10:39:51.804 [main] INFO org.apache.hadoop.hdfs.DFSClient - Could not obtain BP-1464254149-172.17.0.2-1477381671113:blk_1073742576_1752 from any node: java.io.IOException: No live nodes contain block BP-1464254149-172.17.0.2-1477381671113:blk_1073742576_1752 after checking nodes = [172.17.0.2:50010], ignoredNodes = null No live nodes contain current block Block locations: 172.17.0.2:50010 Dead nodes:  172.17.0.2:50010.

找不到任何有效的解决方案。感谢您的帮助。
已编辑:以下是我的主机系统的/etc/hosts的条目(从我调用作业的位置)-

127.0.0.1       localhost    
255.255.255.255 broadcasthost    
::1             localhost    
172.17.0.2 sandbox.hortonworks.com localhost
htrmnn0y

htrmnn0y1#

这是你提到的 URI uri = URI.create ("hdfs://localhost:8020/user/maria_dev/test.txt"); 但它正试图达到 sandbox.hortonworks.com 尝试更改uri

相关问题