org.apache.hadoop.tools.DistCp.run()方法的使用及代码示例

x33g5p2x  于2022-01-18 转载在 其他  
字(4.4k)|赞(0)|评价(0)|浏览(142)

本文整理了Java中org.apache.hadoop.tools.DistCp.run()方法的一些代码示例,展示了DistCp.run()的具体用法。这些代码示例主要来源于Github/Stackoverflow/Maven等平台,是从一些精选项目中提取出来的代码,具有较强的参考意义,能在一定程度帮忙到你。DistCp.run()方法的具体详情如下:
包路径:org.apache.hadoop.tools.DistCp
类名称:DistCp
方法名:run

DistCp.run介绍

[英]Implementation of Tool::run(). Orchestrates the copy of source file(s) to target location, by: 1. Creating a list of files to be copied to target. 2. Launching a Map-only job to copy the files. (Delegates to execute().)
[中]工具::run()的实现。通过以下方式将源文件的副本编排到目标位置:1。创建要复制到目标的文件列表。2.启动仅映射作业以复制文件。(委托执行()

代码示例

代码示例来源:origin: apache/hive

public static boolean runDistCp(List<Path> srcPaths, Path dst, Configuration conf)
  throws IOException {
 DistCpOptions options = new DistCpOptions.Builder(srcPaths, dst)
   .withSyncFolder(true)
   .withCRC(true)
   .preserve(FileAttribute.BLOCKSIZE)
   .build();
 // Creates the command-line parameters for distcp
 List<String> params = constructDistCpParams(srcPaths, dst, conf);
 try {
  conf.setBoolean("mapred.mapper.new-api", true);
  DistCp distcp = new DistCp(conf, options);
  // HIVE-13704 states that we should use run() instead of execute() due to a hadoop known issue
  // added by HADOOP-10459
  if (distcp.run(params.toArray(new String[params.size()])) == 0) {
   return true;
  } else {
   return false;
  }
 } catch (Exception e) {
  throw new IOException("Cannot execute DistCp process: " + e, e);
 } finally {
  conf.setBoolean("mapred.mapper.new-api", false);
 }
}

代码示例来源:origin: apache/hive

@Override
public boolean runDistCp(List<Path> srcPaths, Path dst, Configuration conf) throws IOException {
   DistCpOptions options = new DistCpOptions.Builder(srcPaths, dst)
   .withSyncFolder(true)
   .withCRC(true)
   .preserve(FileAttribute.BLOCKSIZE)
   .build();
 // Creates the command-line parameters for distcp
 List<String> params = constructDistCpParams(srcPaths, dst, conf);
 try {
  conf.setBoolean("mapred.mapper.new-api", true);
  DistCp distcp = new DistCp(conf, options);
  // HIVE-13704 states that we should use run() instead of execute() due to a hadoop known issue
  // added by HADOOP-10459
  if (distcp.run(params.toArray(new String[0])) == 0) {
   return true;
  } else {
   return false;
  }
 } catch (Exception e) {
  throw new IOException("Cannot execute DistCp process: " + e, e);
 } finally {
  conf.setBoolean("mapred.mapper.new-api", false);
 }
}

代码示例来源:origin: airbnb/reair

@Override
 public void run() {
  int ret = distCp.run(options.toArray(new String[] {}));
  retVal.set(Integer.valueOf(ret));
 }
};

代码示例来源:origin: airbnb/reair

@Override
 public void run() {
  int ret = distCp.run(options.toArray(new String[] {}));
  retVal.set(Integer.valueOf(ret));
 }
};

代码示例来源:origin: org.apache.hive.shims/hive-shims-0.23

@Override
public boolean runDistCp(List<Path> srcPaths, Path dst, Configuration conf) throws IOException {
   DistCpOptions options = new DistCpOptions.Builder(srcPaths, dst)
   .withSyncFolder(true)
   .withCRC(true)
   .preserve(FileAttribute.BLOCKSIZE)
   .build();
 // Creates the command-line parameters for distcp
 List<String> params = constructDistCpParams(srcPaths, dst, conf);
 try {
  conf.setBoolean("mapred.mapper.new-api", true);
  DistCp distcp = new DistCp(conf, options);
  // HIVE-13704 states that we should use run() instead of execute() due to a hadoop known issue
  // added by HADOOP-10459
  if (distcp.run(params.toArray(new String[0])) == 0) {
   return true;
  } else {
   return false;
  }
 } catch (Exception e) {
  throw new IOException("Cannot execute DistCp process: " + e, e);
 } finally {
  conf.setBoolean("mapred.mapper.new-api", false);
 }
}

代码示例来源:origin: org.apache.accumulo/accumulo-test

assertEquals("Failed to run distcp: " + Arrays.toString(distCpArgs), 0, cp.run(distCpArgs));

代码示例来源:origin: org.apache.hadoop/hadoop-distcp

/**
 * test methods run end execute of DistCp class. silple copy file
 * @throws Exception 
 */
 @Test
 public void testCleanup() throws Exception {

   Configuration conf = getConf();

   Path stagingDir = JobSubmissionFiles.getStagingDir(new Cluster(conf),
     conf);
   stagingDir.getFileSystem(conf).mkdirs(stagingDir);
   Path soure = createFile("tmp.txt");
   Path target = createFile("target.txt");

   DistCp distcp = new DistCp(conf, null);
   String[] arg = { soure.toString(), target.toString() };

   distcp.run(arg);
   Assert.assertTrue(fs.exists(target));

 
 }

代码示例来源:origin: io.hops/hadoop-distcp

/**
 * test methods run end execute of DistCp class. silple copy file
 * @throws Exception 
 */
 @Test
 public void testCleanup() throws Exception {

   Configuration conf = getConf();

   Path stagingDir = JobSubmissionFiles.getStagingDir(new Cluster(conf),
     conf);
   stagingDir.getFileSystem(conf).mkdirs(stagingDir);
   Path soure = createFile("tmp.txt");
   Path target = createFile("target.txt");

   DistCp distcp = new DistCp(conf, null);
   String[] arg = { soure.toString(), target.toString() };

   distcp.run(arg);
   Assert.assertTrue(fs.exists(target));

 
 }

相关文章