winutils项目的Windows10、Hadoop3.0.0。map reduce作业在ide(intellij idea)中运行良好,但在windows命令行(fat jar)中失败:
java -jar target/app1-1.0-SNAPSHOT-jar-with-dependencies.jar "E://folderin" "E://folderout" -Xmx8g
返回错误:
Exception in thread "main" java.io.IOException: Cannot initialize Cluster. Please check your configuration for mapreduce.framework.name and the correspond server addresses.
at org.apache.hadoop.mapreduce.Cluster.initialize(Cluster.java:116)
在env variables hadoop\u home=c:\hadoop,pom文件中:
<dependencies>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-client</artifactId>
<version>${hadoop.version}</version>
</dependency>
</dependencies>
<build>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-assembly-plugin</artifactId>
<version>3.1.1</version>
<configuration>
<descriptorRefs>
<descriptorRef>jar-with-dependencies</descriptorRef>
</descriptorRefs>
<archive>
<manifest>
<mainClass>com.sample.app1.Starter</mainClass>
<addClasspath>true</addClasspath>
</manifest>
</archive>
</configuration>
<executions>
<execution>
<id>make-assembly</id>
<phase>install</phase>
<goals>
<goal>single</goal>
</goals>
</execution>
</executions>
</plugin>
</plugins>
</build>
</project>
1条答案
按热度按时间kzmpq1sx1#
intellj idea将.m2文件夹放入类路径,这就是idea工作的原因。通过在pom.xml中添加hadoop库解决了此问题: