在windows上构建Hadoop2.2

7vhp5slm  于 2021-06-04  发布在  Hadoop
关注(0)|答案(7)|浏览(505)

我正在试着在windows上安装hadoop,到目前为止还很麻烦。
这是我不断得到的错误

[ERROR] Failed to execute goal org.codehaus.mojo:exec-maven-plugin:1.2:exec (compile-ms-winutils) on project hadoop-common: Command execution failed. Cannot run program "msbuild" (in directory "C:\hdfs\hadoop-common-project\hadoop-common"): CreateProcess error=2, The system cannot find the file specified -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException
[ERROR]
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR]   mvn <goals> -rf :hadoop-common

我已经在Windows7x64框的环境中设置了msbuild.exe、WindowsSDK7.1、cygwin64、maven3.0.5和jdk7。
这是我从apache网站下载的Hadoop2.2.0src。
在系统环境下,我设置了

C:\Windows\Microsoft.NET\Framework64\v4.0.30319

这是msbuild的路径,但仍然不起作用,有人知道如何修复吗?
谢谢
更新:我在hadoop common project\hadoop common中修改了pom.xml

<executable>msbuild</executable>

<executable>msbuild.exe</executable>

现在,我得到以下错误

[ERROR] Failed to execute goal org.codehaus.mojo:exec-maven-plugin:1.2:exec (compile-ms-winutils) on project hadoop-common: Command execution failed. Process exited with an error: 1(Exit value: 1) -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException
[ERROR]
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR]   mvn <goals> -rf :hadoop-common
kzipqqlq

kzipqqlq1#

我也有同样的问题。我通过
打开dos命令提示符
启动“c:\program files(x86)\microsoft visual studio 10.0\vc\vcvarsall.bat”,修改路径变量并设置其他一些变量
之后,我直接从这个命令提示符启动cygwin
c:\cygwin64\bin\mintty.exe-i/cygwin-terminal.ico-
现在路径格式正确,找到了msbuild。。。
希望这有帮助!
顺便说一句:如果它抱怨msbuild的版本错误,可能您必须安装visualstudioexpress2010。或者正如我在其他地方读到的,您可以使用visualstudioexpress2012并将vs100comntools设置为指向vs110comntools。

laximzn5

laximzn52#

我可以通过以下两个步骤解决这个问题。
1) 将msbuild路径(c:\windows\microsoft.net\framework64\v4.0.30319)添加到环境变量path
2) 安装microsoft-visual-studio-2010-professionalhttp://download.cnet.com/microsoft-visual-studio-2010-professional/3000-2212_4-10618634.html
注意:在安装visual-studio-2010-professional之前,我尝试了windows-sdk-7.1,但没有帮助。所以我在安装visual-studio-2010-professional之前卸载了windows-sdk-7.1。我为visual-studio-2010-professional提供的链接也是一个试用版。

eufgjt7s

eufgjt7s3#

似乎路径有问题。。
将msbuild路径添加到环境变量path

7y4bm7vi

7y4bm7vi4#

我按照以下步骤安装了hadoop2.2.0
为windows构建hadoop bin分发的步骤
下载并安装microsoft windows sdk v7.1。
下载并安装unix命令行工具cygwin。
下载并安装maven 3.1.1。
下载ProtocolBuffers2.5.0并解压缩到一个文件夹(比如c:\protobuf)。
添加环境变量java\u home、m2\u home和platform(如果尚未添加)。注意:变量名平台区分大小写。对于在64位或32位系统上构建,值将为x64或win32。编辑path变量以添加cygwin的bin目录(例如c:\cygwin64\bin)、maven的bin目录(例如c:\maven\bin)和协议缓冲区的安装路径(例如c:\protobuf)。
下载hadoop-2.2.0-src.tar.gz并解压缩到具有短路径的文件夹(例如c:\hdfs),以避免由于windows中的最大路径长度限制而导致的运行时问题。
选择开始-->所有程序-->microsoft windows sdk v7.1并打开windows sdk 7.1命令提示符。将目录更改为hadoop源代码文件夹(c:\hdfs)。使用选项-pdist、native win-dskiptests-dtar执行mvn包以创建windows二进制tar分发。
如果上一步一切顺利,那么本机发行版hadoop-2.2.0.tar.gz将在c:\hdfs\hadoop dist\target\hadoop-2.2.0目录中创建。
安装hadoop
将hadoop-2.2.0.tar.gz解压缩到一个文件夹(比如c:\hadoop)。
添加环境变量hadoop\u home并编辑path变量以添加hadoop\u home的bin目录(例如c:\hadoop\bin)。
配置hadoop
c:\hadoop\etc\hadoop\core-site.xml

<configuration>
        <property>
                <name>fs.defaultFS</name>
                <value>hdfs://localhost:9000</value>
        </property>
</configuration>

c:\hadoop\etc\hadoop\hdfs-site.xml

<configuration>
        <property>
                <name>dfs.replication</name>
                <value>1</value>
        </property>
        <property>
                <name>dfs.namenode.name.dir</name>
                <value>file:/hadoop/data/dfs/namenode</value>
        </property>
        <property>
                <name>dfs.datanode.data.dir</name>
                <value>file:/hadoop/data/dfs/datanode</value>
        </property>
</configuration>

c:\hadoop\etc\hadoop\mapred-site.xml

<configuration>
        <property>
           <name>mapreduce.framework.name</name>
           <value>yarn</value>
        </property>
</configuration>

c:\hadoop\etc\hadoop\yarn-site.xml

<configuration>
        <property>
           <name>yarn.nodemanager.aux-services</name>
           <value>mapreduce_shuffle</value>
        </property>
        <property>
           <name>yarn.nodemanager.aux-services.mapreduce_shuffle.class</name>
           <value>org.apache.hadoop.mapred.ShuffleHandler</value>
        </property>
</configuration>

设置名称节点格式
namenode第一次需要格式化。

C:\Users\abhijitg>cd c:\hadoop\bin 
c:\hadoop\bin>hdfs namenode –format

启动hdfs(namenode和datanode)

C:\Users\abhijitg>cd c:\hadoop\sbin
c:\hadoop\sbin>start-dfs

启动mapreduce(资源管理器和节点管理器)

C:\Users\abhijitg>cd c:\hadoop\sbin
c:\hadoop\sbin>start-yarn
starting yarn daemons

总共四个单独的命令提示符窗口将自动打开以运行namenode、datanode、resource manager、node manager
参考:在microsoft windows操作系统中构建、安装、配置和运行apache hadoop 2.2.0

ljo96ir5

ljo96ir55#

我找到了这个问题的解决办法。我在Windows8.1上安装了VS2012,并且面临着这个问题。我在hadoop common下为该执行id checkout pom.xml并运行以下命令

F:\HDP>msbuild F:\HDP\hadoop-common-project\hadoop-common\src\main\winutils\winutils.sln /nologo/p:Configuration=Release;OutDir=bin

它给了我以下的例外

C:\Program Files(x86)\MSBuild\Microsoft.Cpp\v4.0\V110\Microsoft.Cpp.Platform.targets(42,5): error MSB8020: The builds tools for Visual Studio 2010 (Platform Toolset ='v100') cannot be found.
To build using the v100 build tools, either click the Project menu or right-click the solution, and then select "Update VC++ Projects...". Install Visual Studio 2010 to build using the Visual Studio 2010 build tools.
[F:\HDP\hadoop-common-project\hadoop-common\src\main\winutils\libwinutils.vcxproj]

因为我的机器上没有visualstudio2010,所以问题必须在解决方案中。我在vs 2012中打开了这个解决方案并编译了一次。再次执行maven给了我另一个错误,但这次指向hadoop common\src\main\native\native.sln。我在vs 2012上也打开了,没有错误。
希望这有帮助
谢谢罗希特

kmpatx3s

kmpatx3s6#

您可能需要cmake.exe-使用官方链接获取http://www.cmake.org/download/
打开hadoop-2.6.0-src.tar.gz或您的版本中的building.txt文件。
它告诉您如何为windows构建-请检查所有这些:
my building.txt说:

Building on Windows
Requirements:

* Windows System
* JDK 1.6+
* Maven 3.0 or later
* Findbugs 1.3.9 (if running findbugs)
* ProtocolBuffer 2.5.0
* CMake 2.6 or newer
* Windows SDK or Visual Studio 2010 Professional
* Unix command-line tools from GnuWin32 or Cygwin: sh, mkdir, rm, cp, tar, gzip
* zlib headers (if building native code bindings for zlib)
* Internet connection for first build (to fetch all Maven and Hadoop dependencies)

If using Visual Studio, it must be Visual Studio 2010 Professional (not 2012).
Do not use Visual Studio Express.  It does not support compiling for 64-bit,
which is problematic if running a 64-bit system.  The Windows SDK is free to
download here:

http://www.microsoft.com/en-us/download/details.aspx?id=8279
tcomlyy6

tcomlyy67#

即使在完成了文档和此处提到的所有步骤之后,我仍然出现以下错误:

Failed to execute goal org.codehaus.mojo:exec-maven-plugin:1.3.1:exec 
(compile-ms-winutils) on project hadoop-common

详细的错误是:

You are attempting to build a Win32 application from an x64 environment. 
If using the Windows 7.1 SDK build environment, type setenv /x86

所以我添加了以下环境变量,解决了这个问题

Platform: x64

相关问题