我尝试使用hadoopapi来使用distcp类进行分布式复制,但是在尝试连接到集群时它会抛出一个错误。我试图更改hadoop和hdfs的配置文件,但似乎不起作用。我正在最新的cloudera快速启动上测试这个应用程序
我运行这个命令来执行这个类。java-cp myjar com.keedio.hadoop.mover
包com.keedio.hadoop;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.*;
import org.apache.hadoop.tools.DistCp;
import org.apache.hadoop.tools.DistCpOptionSwitch;
import org.apache.hadoop.tools.DistCpOptions;
import org.apache.hadoop.util.ToolRunner;
import java.util.Collections;
public class Mover {
public static void main(String args []) {
try {
FileSystem fs = FileSystem.getLocal(new Configuration());
FileSystem fs2= FileSystem.get(java.net.URI.create("file:///"),new
Configuration());
DistCpOptions distCpOptions=new DistCpOptions(new Path("file:/Users/jvasquez/Desktop/ficheros1"),new Path("file:/Users/jvasquez/Desktop/ficheros2"));
String argumentos [] = {"file:////Users/jvasquez/Desktop/ficheros1","file:///Users/jvasquez/Desktop/ficheros2"};
Configuration conf=new Configuration();
conf.addResource(new org.apache.hadoop.fs.Path("/etc/hadoop/conf/yarn-site.xml"));
conf.addResource(new org.apache.hadoop.fs.Path("/etc/hadoop/conf/core-site.xml"));
conf.addResource(new org.apache.hadoop.fs.Path("/etc/hadoop/conf/mapred-site.xml"));
conf.addResource(new Path("/etc/hadoop/conf/hdfs-site.xml"));
DistCp distCp= new DistCp(conf,distCpOptions);
ToolRunner.run(distCp,argumentos);
}
catch (Exception e) {
e.printStackTrace();
}
}
}
这就是错误
log4j:WARN No appenders could be found for logger
(org.apache.hadoop.metrics2.lib.MutableMetricsFactory).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
java.io.IOException: Cannot initialize Cluster. Please check your configuration for mapreduce.framework.name and the correspond server addresses.
at org.apache.hadoop.mapreduce.Cluster.initialize(Cluster.java:143)
at org.apache.hadoop.mapreduce.Cluster.<init>(Cluster.java:108)
at org.apache.hadoop.mapreduce.Cluster.<init>(Cluster.java:101)
at org.apache.hadoop.tools.DistCp.createMetaFolderPath(DistCp.java:471)
at org.apache.hadoop.tools.DistCp.<init>(DistCp.java:107)
at com.keedio.hadoop.Mover.main(Mover.java:48)
1条答案
按热度按时间mnowg1ta1#
最后我解决了这个问题,结果是我不得不使用命令hadoopjar将应用程序作为mapreduce应用程序启动。