hadoop—使用java从hdfs中的一个目录复制到hdfs中的另一个目录

xxb16uws  于 2021-05-29  发布在  Hadoop
关注(0)|答案(1)|浏览(271)

我正在尝试将数据从hdfs中的一个目录复制到hdfs中的另一个目录，但是我面临的问题很少。这是我的代码片段。

Configuration conf = new Configuration();
  FileSystem fs = FileSystem.get(conf);
  LOGGER.info("Connected");
  Path source=new Path("/data_dev/deepak/src/raw/epic/cl_qanswer_qa/hdp_process_date=2017-07-25/hour=00/minute=00/");
  Path target=new Path("/data_dev/deepak/dest/raw/epics/cl_qanswer_qa/hdp_process_date=2017-07-25/hour=00/minute=00/");
  System.out.println(source);
  System.out.println(target);
  System.out.println("source"+fs.exists(source));
  System.out.println("source"+fs.exists(target));

  FileSystem srcfs = FileSystem.get(conf);
  FileSystem dstFS = FileSystem.get(conf);

  RemoteIterator<LocatedFileStatus> sourceFiles = srcfs.listFiles(source, false);
  LOGGER.info(sourceFiles.toString());
  LOGGER.info("source File System "+fs.toString());
  LOGGER.info("destniation File System"+dstFS.toString());
  if(!fs.exists(target))
  {
      fs.create(target);
      LOGGER.info("created thr path");
  }

  if(sourceFiles != null) {
      while(sourceFiles.hasNext()){
          System.out.println(sourceFiles.toString());
          Path srcfilepath = sourceFiles.next().getPath();
          System.out.println(srcfilepath);
          if(FileUtil.copy(srcfs, srcfilepath, dstFS, target, false,true, conf)){
              System.out.println("Copied Successfully" );
          }
          else
          {
              System.out.println("Copy Failed");
          }
      }           
  }
  srcfs.close();
  dstFS.close();
  fs.close();   
}

如果目标目录不存在，那么我将在上面的代码中创建目标目录。因此，只有当目标目录不在那里时，我才会得到这个错误。

hadoop jar Moving.jar
Dec 10, 2017 6:07:30 PM com.ghs.misc.Moving main
INFO: Connected
/data_dev/deepak/src/raw/epic/cl_qanswer_qa/hdp_process_date=2017-07-25/hour=00/minute=00
/data_dev/deepak/dest/raw/epics/cl_qanswer_qa/hdp_process_date=2017-07-25/hour=00/minute=00
sourcetrue
sourcefalse
Dec 10, 2017 6:07:30 PM com.ghs.misc.Moving main
INFO: org.apache.hadoop.fs.FileSystem$6@29a1c0b7
Dec 10, 2017 6:07:30 PM com.ghs.misc.Moving main
INFO: source File System 
DFS[DFSClient[clientName=DFSClient_NONMAPREDUCE_61931562_1(auth:KERBEROS)]]
Dec 10, 2017 6:07:30 PM com.ghs.misc.Moving main
INFO: destniation File 
SystemDFS[DFSClient[clientName=DFSClient_NONMAPREDUCE_61931562_1 (auth:KERBEROS)]]
Dec 10, 2017 6:07:30 PM com.ghs.misc.Moving main
INFO: created thr path
org.apache.hadoop.fs.FileSystem$6@29a1c0b7
/data_dev/deepak/src/raw/epic/cl_qanswer_qa/hdp_process_date=2017-07-25/hour=00/minute=00/HQAQA.lzo
Copied Successfully
org.apache.hadoop.fs.FileSystem$6@29a1c0b7
/data_dev/deepak/src/raw/epic/cl_qanswer_qa/hdp_process_date=2017-07-25/hour=00/minute=00/HQAQA.lzo.index
Copied Successfully
org.apache.hadoop.fs.FileSystem$6@29a1c0b7
/data_dev/deepak/src/raw/epic/cl_qanswer_qa/hdp_process_date=2017-07-25/hour=00/minute=00/Test1.txt
Copied Successfully
org.apache.hadoop.fs.FileSystem$6@29a1c0b7
 /data_dev/deepak/src/raw/epic/cl_qanswer_qa/hdp_process_date=2017-07-25/hour=00/minute=00/Test2.txt
Copied Successfully
org.apache.hadoop.fs.FileSystem$6@29a1c0b7
/data_dev/deepak/src/raw/epic/cl_qanswer_qa/hdp_process_date=2017-07-25/hour=00/minute=00/Test3.txt
Copied Successfully17/12/10 
18:07:34 ERROR hdfs.DFSClient: Failed to close inode 364006128
 org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.hdfs.server.namenode.LeaseExpiredException): No lease on /data_dev/deepak/dest/raw/epics/cl_qanswer_qa/hdp_process_date=2017-07-25/hour=00/minute=00 (inode 364006128): File does not exist. Holder DFSClient_NONMAPREDUCE_61931562_1 does not have any open files.
    at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkLease(FSNamesystem.java:3693)
    at     org.apache.hadoop.hdfs.server.namenode.FSNamesystem.completeFileInternal(FSNamesystem.java:3781)
    at     org.apache.hadoop.hdfs.server.namenode.FSNamesystem.completeFile(FSNamesystem.java:3748)
    at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.complete(NameNodeRpcServer.java:912)
    at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.complete(ClientNamenodeProtocolServerSideTranslatorPB.java:549)
    at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
    at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:640)
    at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:982)
    at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2351)
    at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2347)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:422)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1866)
    at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2347)

    at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1554)
    at org.apache.hadoop.ipc.Client.call(Client.java:1498)
    at org.apache.hadoop.ipc.Client.call(Client.java:1398)
    at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:233)
    at com.sun.proxy.$Proxy10.complete(Unknown Source)
    at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.complete(ClientNamenodeProtocolTranslatorPB.java:503)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:291)
    at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:203)
    at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:185)
    at com.sun.proxy.$Proxy11.complete(Unknown Source)
    at org.apache.hadoop.hdfs.DFSOutputStream.completeFile(DFSOutputStream.java:2496)
    at org.apache.hadoop.hdfs.DFSOutputStream.closeImpl(DFSOutputStream.java:2472)
    at org.apache.hadoop.hdfs.DFSOutputStream.close(DFSOutputStream.java:2437)
    at org.apache.hadoop.hdfs.DFSClient.closeAllFilesBeingWritten(DFSClient.java:949)
    at org.apache.hadoop.hdfs.DFSClient.closeOutputStreams(DFSClient.java:981)
    at org.apache.hadoop.hdfs.DistributedFileSystem.close(DistributedFileSystem.java:1211)
    at com.ghs.misc.Moving.main(Moving.java:67)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.apache.hadoop.util.RunJar.run(RunJar.java:233)
    at org.apache.hadoop.util.RunJar.main(RunJar.java:148)

任何帮助都将不胜感激。提前谢谢！

Javahadoophdfsbigdatafile-handling

1条答案

按热度按时间
pengsaosao

pengsaosao1#

我发现了错误。创建目录时，我使用filesystem.create（）创建目录。但实际上它是在指定的路径上创建一个fsdataoutputstream。所以我把它改成filesystem.mkdirs（targetpath），它解决了我的错误。现在我的代码运行良好。抱歉，我犯了个愚蠢的错误，浪费了你们的时间。

if(!fs.exists(target))
{
  fs.mkdirs(target);      //I have used this ->fs.create(target);
  LOGGER.info("created the path");
}
赞(0）分享  回复(0）举报  2021-05-29
我来回答

相关问题

    查看更多

    热门标签

    更多
    JavaquerypythonNode开发语言requestUtil数据库Table后端算法LoggerMessageElementParser

    最新问答

    更多
    • 蜀ICP备2022004421号-2 NULL123 https://www.null123.com © All rights reserved
    • 本站内容来源互联网,如果侵犯您的权益请联系我们删除, 联系方式：448109455@qq.com