本文整理了Java中org.apache.hadoop.tools.DistCp.run()
方法的一些代码示例,展示了DistCp.run()
的具体用法。这些代码示例主要来源于Github
/Stackoverflow
/Maven
等平台,是从一些精选项目中提取出来的代码,具有较强的参考意义,能在一定程度帮忙到你。DistCp.run()
方法的具体详情如下:
包路径:org.apache.hadoop.tools.DistCp
类名称:DistCp
方法名:run
[英]Implementation of Tool::run(). Orchestrates the copy of source file(s) to target location, by: 1. Creating a list of files to be copied to target. 2. Launching a Map-only job to copy the files. (Delegates to execute().)
[中]工具::run()的实现。通过以下方式将源文件的副本编排到目标位置:1。创建要复制到目标的文件列表。2.启动仅映射作业以复制文件。(委托执行()
代码示例来源:origin: apache/hive
public static boolean runDistCp(List<Path> srcPaths, Path dst, Configuration conf)
throws IOException {
DistCpOptions options = new DistCpOptions.Builder(srcPaths, dst)
.withSyncFolder(true)
.withCRC(true)
.preserve(FileAttribute.BLOCKSIZE)
.build();
// Creates the command-line parameters for distcp
List<String> params = constructDistCpParams(srcPaths, dst, conf);
try {
conf.setBoolean("mapred.mapper.new-api", true);
DistCp distcp = new DistCp(conf, options);
// HIVE-13704 states that we should use run() instead of execute() due to a hadoop known issue
// added by HADOOP-10459
if (distcp.run(params.toArray(new String[params.size()])) == 0) {
return true;
} else {
return false;
}
} catch (Exception e) {
throw new IOException("Cannot execute DistCp process: " + e, e);
} finally {
conf.setBoolean("mapred.mapper.new-api", false);
}
}
代码示例来源:origin: apache/hive
@Override
public boolean runDistCp(List<Path> srcPaths, Path dst, Configuration conf) throws IOException {
DistCpOptions options = new DistCpOptions.Builder(srcPaths, dst)
.withSyncFolder(true)
.withCRC(true)
.preserve(FileAttribute.BLOCKSIZE)
.build();
// Creates the command-line parameters for distcp
List<String> params = constructDistCpParams(srcPaths, dst, conf);
try {
conf.setBoolean("mapred.mapper.new-api", true);
DistCp distcp = new DistCp(conf, options);
// HIVE-13704 states that we should use run() instead of execute() due to a hadoop known issue
// added by HADOOP-10459
if (distcp.run(params.toArray(new String[0])) == 0) {
return true;
} else {
return false;
}
} catch (Exception e) {
throw new IOException("Cannot execute DistCp process: " + e, e);
} finally {
conf.setBoolean("mapred.mapper.new-api", false);
}
}
代码示例来源:origin: airbnb/reair
@Override
public void run() {
int ret = distCp.run(options.toArray(new String[] {}));
retVal.set(Integer.valueOf(ret));
}
};
代码示例来源:origin: airbnb/reair
@Override
public void run() {
int ret = distCp.run(options.toArray(new String[] {}));
retVal.set(Integer.valueOf(ret));
}
};
代码示例来源:origin: org.apache.hive.shims/hive-shims-0.23
@Override
public boolean runDistCp(List<Path> srcPaths, Path dst, Configuration conf) throws IOException {
DistCpOptions options = new DistCpOptions.Builder(srcPaths, dst)
.withSyncFolder(true)
.withCRC(true)
.preserve(FileAttribute.BLOCKSIZE)
.build();
// Creates the command-line parameters for distcp
List<String> params = constructDistCpParams(srcPaths, dst, conf);
try {
conf.setBoolean("mapred.mapper.new-api", true);
DistCp distcp = new DistCp(conf, options);
// HIVE-13704 states that we should use run() instead of execute() due to a hadoop known issue
// added by HADOOP-10459
if (distcp.run(params.toArray(new String[0])) == 0) {
return true;
} else {
return false;
}
} catch (Exception e) {
throw new IOException("Cannot execute DistCp process: " + e, e);
} finally {
conf.setBoolean("mapred.mapper.new-api", false);
}
}
代码示例来源:origin: org.apache.accumulo/accumulo-test
assertEquals("Failed to run distcp: " + Arrays.toString(distCpArgs), 0, cp.run(distCpArgs));
代码示例来源:origin: org.apache.hadoop/hadoop-distcp
/**
* test methods run end execute of DistCp class. silple copy file
* @throws Exception
*/
@Test
public void testCleanup() throws Exception {
Configuration conf = getConf();
Path stagingDir = JobSubmissionFiles.getStagingDir(new Cluster(conf),
conf);
stagingDir.getFileSystem(conf).mkdirs(stagingDir);
Path soure = createFile("tmp.txt");
Path target = createFile("target.txt");
DistCp distcp = new DistCp(conf, null);
String[] arg = { soure.toString(), target.toString() };
distcp.run(arg);
Assert.assertTrue(fs.exists(target));
}
代码示例来源:origin: io.hops/hadoop-distcp
/**
* test methods run end execute of DistCp class. silple copy file
* @throws Exception
*/
@Test
public void testCleanup() throws Exception {
Configuration conf = getConf();
Path stagingDir = JobSubmissionFiles.getStagingDir(new Cluster(conf),
conf);
stagingDir.getFileSystem(conf).mkdirs(stagingDir);
Path soure = createFile("tmp.txt");
Path target = createFile("target.txt");
DistCp distcp = new DistCp(conf, null);
String[] arg = { soure.toString(), target.toString() };
distcp.run(arg);
Assert.assertTrue(fs.exists(target));
}
内容来源于网络,如有侵权,请联系作者删除!