在osx中构建hadoop3.1.1以获取本机库

l5tcr1uw  于 2021-05-27  发布在  Hadoop
关注(0)|答案(1)|浏览(365)

我安装hadoop brew install hadoop 然后使用 pip install pyarrow 作为客户

client = pa.hdfs.connect('localhost', 9000, driver='libhdfs')

for x in client.ls('/'):
    print(x)

获取错误

Traceback (most recent call last):
  File "/Users/wyx/project/py3.7aio/hdfs/list_dir.py", line 13, in <module>
    client = pa.hdfs.connect('localhost', 9000, )
  File "/Users/wyx/project/py3.7aio/.env/lib/python3.6/site-packages/pyarrow/hdfs.py", line 207, in connect
    extra_conf=extra_conf)
  File "/Users/wyx/project/py3.7aio/.env/lib/python3.6/site-packages/pyarrow/hdfs.py", line 38, in __init__
    self._connect(host, port, user, kerb_ticket, driver, extra_conf)
  File "pyarrow/io-hdfs.pxi", line 89, in pyarrow.lib.HadoopFileSystem._connect
  File "pyarrow/error.pxi", line 83, in pyarrow.lib.check_status
pyarrow.lib.ArrowIOError: Unable to load libhdfs

如何获取libhdfs或libhdfs3
我想这可能是关于本Map书馆的，因为

➜  hadoop-rel-release-3.1.1 hadoop checknative -a
2019-02-21 15:46:20,805 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Native library checking:
hadoop:  false
zlib:    false
zstd  :  false
snappy:  false
lz4:     false
bzip2:   false
openssl: false
ISA-L:   false
2019-02-21 15:46:20,934 INFO util.ExitUtil: Exiting with status 1: ExitException

在我之后 brew install gcc autoconf automake libtool cmake snappy gzip bzip2 protobuf@2.5 zlib openssl 下载 hadoop-rel-release-3.1.1 建造。

[INFO] Compiling 3 source files to /Users/wyx/Downloads/hadoop-rel-release-3.1.1/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/target/classes
[INFO]
[INFO] --- hadoop-maven-plugins:3.1.1:cmake-compile (cmake-compile) @ hadoop-yarn-server-nodemanager ---
[DEBUG] Configuring mojo org.apache.hadoop:hadoop-maven-plugins:3.1.1:cmake-compile from plugin realm ClassRealm[plugin>org.apache.hadoop:hadoop-maven-plugins:3.1.1, parent: sun.misc.Launcher$AppClassLoader@55f96302]
[DEBUG] Configuring mojo 'org.apache.hadoop:hadoop-maven-plugins:3.1.1:cmake-compile' with basic configurator -->
[DEBUG]   (f) env = {CFLAGS=null}
[DEBUG]   (f) output = /Users/wyx/Downloads/hadoop-rel-release-3.1.1/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/target/native
[DEBUG]   (f) source = /Users/wyx/Downloads/hadoop-rel-release-3.1.1/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src
[DEBUG]   (f) vars = {HADOOP_CONF_DIR=../etc/hadoop, JVM_ARCH_DATA_MODEL=64}
[DEBUG] -- end configuration --
[INFO] Running cmake /Users/wyx/Downloads/hadoop-rel-release-3.1.1/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src -DHADOOP_CONF_DIR=../etc/hadoop -DJVM_ARCH_DATA_MODEL=64 -G Unix Makefiles
[INFO] with extra environment variables {
  CFLAGS = ''
}
[INFO] Running make -j 8 VERBOSE=1
[WARNING] /usr/local/Cellar/cmake/3.13.4/bin/cmake -S/Users/wyx/Downloads/hadoop-rel-release-3.1.1/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src -B/Users/wyx/Downloads/hadoop-rel-release-3.1.1/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/target/native --check-build-system CMakeFiles/Makefile.cmake 0
[WARNING] /usr/local/Cellar/cmake/3.13.4/bin/cmake -E cmake_progress_start /Users/wyx/Downloads/hadoop-rel-release-3.1.1/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/target/native/CMakeFiles /Users/wyx/Downloads/hadoop-rel-release-3.1.1/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/target/native/CMakeFiles/progress.marks
[WARNING] /Applications/Xcode.app/Contents/Developer/usr/bin/make -f CMakeFiles/Makefile2 all
[WARNING] /Applications/Xcode.app/Contents/Developer/usr/bin/make -f CMakeFiles/container.dir/build.make CMakeFiles/container.dir/depend
[WARNING] /Applications/Xcode.app/Contents/Developer/usr/bin/make -f CMakeFiles/gtest.dir/build.make CMakeFiles/gtest.dir/depend
[WARNING] cd /Users/wyx/Downloads/hadoop-rel-release-3.1.1/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/target/native && /usr/local/Cellar/cmake/3.13.4/bin/cmake -E cmake_depends "Unix Makefiles" /Users/wyx/Downloads/hadoop-rel-release-3.1.1/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src /Users/wyx/Downloads/hadoop-rel-release-3.1.1/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src /Users/wyx/Downloads/hadoop-rel-release-3.1.1/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/target/native /Users/wyx/Downloads/hadoop-rel-release-3.1.1/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/target/native /Users/wyx/Downloads/hadoop-rel-release-3.1.1/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/target/native/CMakeFiles/gtest.dir/DependInfo.cmake --color=
[WARNING] cd /Users/wyx/Downloads/hadoop-rel-release-3.1.1/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/target/native && /usr/local/Cellar/cmake/3.13.4/bin/cmake -E cmake_depends "Unix Makefiles" /Users/wyx/Downloads/hadoop-rel-release-3.1.1/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src /Users/wyx/Downloads/hadoop-rel-release-3.1.1/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src /Users/wyx/Downloads/hadoop-rel-release-3.1.1/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/target/native /Users/wyx/Downloads/hadoop-rel-release-3.1.1/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/target/native /Users/wyx/Downloads/hadoop-rel-release-3.1.1/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/target/native/CMakeFiles/container.dir/DependInfo.cmake --color=
[WARNING] /Applications/Xcode.app/Contents/Developer/usr/bin/make -f CMakeFiles/gtest.dir/build.make CMakeFiles/gtest.dir/build
[WARNING] /Applications/Xcode.app/Contents/Developer/usr/bin/make -f CMakeFiles/container.dir/build.make CMakeFiles/container.dir/build
[WARNING] make[2]: Nothing to be done for `CMakeFiles/gtest.dir/build'.
[WARNING] make[2]: Nothing to be done for `CMakeFiles/container.dir/build'.
[WARNING] [  6%] Built target gtest
[WARNING] [ 46%] Built target container
[WARNING] /Applications/Xcode.app/Contents/Developer/usr/bin/make -f CMakeFiles/cetest.dir/build.make CMakeFiles/cetest.dir/depend
[WARNING] /Applications/Xcode.app/Contents/Developer/usr/bin/make -f CMakeFiles/test-container-executor.dir/build.make CMakeFiles/test-container-executor.dir/depend
[WARNING] /Applications/Xcode.app/Contents/Developer/usr/bin/make -f CMakeFiles/container-executor.dir/build.make CMakeFiles/container-executor.dir/depend
[WARNING] cd /Users/wyx/Downloads/hadoop-rel-release-3.1.1/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/target/native && /usr/local/Cellar/cmake/3.13.4/bin/cmake -E cmake_depends "Unix Makefiles" /Users/wyx/Downloads/hadoop-rel-release-3.1.1/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src /Users/wyx/Downloads/hadoop-rel-release-3.1.1/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src /Users/wyx/Downloads/hadoop-rel-release-3.1.1/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/target/native /Users/wyx/Downloads/hadoop-rel-release-3.1.1/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/target/native /Users/wyx/Downloads/hadoop-rel-release-3.1.1/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/target/native/CMakeFiles/test-container-executor.dir/DependInfo.cmake --color=
[WARNING] cd /Users/wyx/Downloads/hadoop-rel-release-3.1.1/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/target/native && /usr/local/Cellar/cmake/3.13.4/bin/cmake -E cmake_depends "Unix Makefiles" /Users/wyx/Downloads/hadoop-rel-release-3.1.1/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src /Users/wyx/Downloads/hadoop-rel-release-3.1.1/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src /Users/wyx/Downloads/hadoop-rel-release-3.1.1/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/target/native /Users/wyx/Downloads/hadoop-rel-release-3.1.1/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/target/native /Users/wyx/Downloads/hadoop-rel-release-3.1.1/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/target/native/CMakeFiles/container-executor.dir/DependInfo.cmake --color=
[WARNING] cd /Users/wyx/Downloads/hadoop-rel-release-3.1.1/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/target/native && /usr/local/Cellar/cmake/3.13.4/bin/cmake -E cmake_depends "Unix Makefiles" /Users/wyx/Downloads/hadoop-rel-release-3.1.1/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src /Users/wyx/Downloads/hadoop-rel-release-3.1.1/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src /Users/wyx/Downloads/hadoop-rel-release-3.1.1/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/target/native /Users/wyx/Downloads/hadoop-rel-release-3.1.1/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/target/native /Users/wyx/Downloads/hadoop-rel-release-3.1.1/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/target/native/CMakeFiles/cetest.dir/DependInfo.cmake --color=
[WARNING] /Applications/Xcode.app/Contents/Developer/usr/bin/make -f CMakeFiles/container-executor.dir/build.make CMakeFiles/container-executor.dir/build
[WARNING] /Applications/Xcode.app/Contents/Developer/usr/bin/make -f CMakeFiles/test-container-executor.dir/build.make CMakeFiles/test-container-executor.dir/build
[WARNING] /Applications/Xcode.app/Contents/Developer/usr/bin/make -f CMakeFiles/cetest.dir/build.make CMakeFiles/cetest.dir/build
[WARNING] make[2]: Nothing to be done for `CMakeFiles/container-executor.dir/build'.
[WARNING] make[2]: Nothing to be done for `CMakeFiles/test-container-executor.dir/build'.
[WARNING] [ 60%] Built target container-executor
[WARNING] [ 60%] Built target test-container-executor
[WARNING] [ 63%] Building CXX object CMakeFiles/cetest.dir/main/native/container-executor/test/utils/test_docker_util.cc.o
[WARNING] /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/c++   -I/System/Library/Frameworks -I/Users/wyx/Downloads/hadoop-rel-release-3.1.1/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src -I/Users/wyx/Downloads/hadoop-rel-release-3.1.1/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/target/native -I/Users/wyx/Downloads/hadoop-rel-release-3.1.1/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor -I/Users/wyx/Downloads/hadoop-rel-release-3.1.1/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl -isystem /Users/wyx/Downloads/hadoop-rel-release-3.1.1/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/../../../../../hadoop-common-project/hadoop-common/src/main/native/gtest/include  -g -O2 -Wall -pthread -D_FILE_OFFSET_BITS=64 -isysroot /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.14.sdk   -o CMakeFiles/cetest.dir/main/native/container-executor/test/utils/test_docker_util.cc.o -c /Users/wyx/Downloads/hadoop-rel-release-3.1.1/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/test/utils/test_docker_util.cc
[WARNING] In file included from /Users/wyx/Downloads/hadoop-rel-release-3.1.1/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/test/utils/test_docker_util.cc:24:
[WARNING] /Users/wyx/Downloads/hadoop-rel-release-3.1.1/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/utils/docker-util.c:56:13: warning: comparison of array 'args->data' equal to a null pointer is always false [-Wtautological-pointer-compare]
[WARNING]   if (args->data == NULL || args->length >= DOCKER_ARG_MAX) {
[WARNING]       ~~~~~~^~~~    ~~~~
[WARNING] /Users/wyx/Downloads/hadoop-rel-release-3.1.1/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/utils/docker-util.c:63:13: warning: comparison of array 'args->data' not equal to a null pointer is always true [-Wtautological-pointer-compare]
[WARNING]   if (args->data != NULL) {
[WARNING]       ~~~~~~^~~~    ~~~~
[WARNING] /Users/wyx/Downloads/hadoop-rel-release-3.1.1/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/utils/docker-util.c:1227:12: error: no matching function for call to 'getgrouplist'
[WARNING]   int rc = getgrouplist(user, pw->pw_gid, groups, &ngroups);
[WARNING]            ^~~~~~~~~~~~
[WARNING] /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.14.sdk/usr/include/unistd.h:653:6: note: candidate function not viable: no known conversion from 'gid_t *' (aka 'unsigned int *') to 'int *' for 3rd argument
[WARNING] int      getgrouplist(const char *, int, int *, int *);
[WARNING]          ^
[WARNING] In file included from /Users/wyx/Downloads/hadoop-rel-release-3.1.1/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/test/utils/test_docker_util.cc:24:
[WARNING] /Users/wyx/Downloads/hadoop-rel-release-3.1.1/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/utils/docker-util.c:1234:9: error: no matching function for call to 'getgrouplist'
[WARNING]     if (getgrouplist(user, pw->pw_gid, groups, &ngroups) == -1) {
[WARNING]         ^~~~~~~~~~~~
[WARNING] /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.14.sdk/usr/include/unistd.h:653:6: note: candidate function not viable: no known conversion from 'gid_t *' (aka 'unsigned int *') to 'int *' for 3rd argument
[WARNING] int      getgrouplist(const char *, int, int *, int *);
[WARNING]          ^
[WARNING] 2 warnings and 2 errors generated.
[WARNING] make[2]:***[CMakeFiles/cetest.dir/main/native/container-executor/test/utils/test_docker_util.cc.o] Error 1
[WARNING] make[1]:***[CMakeFiles/cetest.dir/all] Error 2
[WARNING] make:***[all] Error 2
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary for Apache Hadoop Main 3.1.1:
[INFO]
[INFO] Apache Hadoop Main ................................. SUCCESS [  1.115 s]
[INFO] Apache Hadoop Build Tools .......................... SUCCESS [  1.784 s]
[INFO] Apache Hadoop Project POM .......................... SUCCESS [  0.991 s]
[INFO] Apache Hadoop Annotations .......................... SUCCESS [  2.055 s]
[INFO] Apache Hadoop Assemblies ........................... SUCCESS [  0.579 s]
[INFO] Apache Hadoop Project Dist POM ..................... SUCCESS [  1.898 s]
[INFO] Apache Hadoop Maven Plugins ........................ SUCCESS [  3.680 s]
[INFO] Apache Hadoop MiniKDC .............................. SUCCESS [  1.903 s]
[INFO] Apache Hadoop Auth ................................. SUCCESS [  4.898 s]
[INFO] Apache Hadoop Auth Examples ........................ SUCCESS [  2.586 s]
[INFO] Apache Hadoop Common ............................... SUCCESS [ 47.036 s]
[INFO] Apache Hadoop NFS .................................. SUCCESS [  4.246 s]
[INFO] Apache Hadoop KMS .................................. SUCCESS [  5.686 s]
[INFO] Apache Hadoop Common Project ....................... SUCCESS [  0.385 s]
[INFO] Apache Hadoop HDFS Client .......................... SUCCESS [ 16.175 s]
[INFO] Apache Hadoop HDFS ................................. SUCCESS [ 36.668 s]
[INFO] Apache Hadoop HDFS Native Client ................... SUCCESS [  3.188 s]
[INFO] Apache Hadoop HttpFS ............................... SUCCESS [  6.474 s]
[INFO] Apache Hadoop HDFS-NFS ............................. SUCCESS [  2.781 s]
[INFO] Apache Hadoop HDFS-RBF ............................. SUCCESS [ 13.786 s]
[INFO] Apache Hadoop HDFS Project ......................... SUCCESS [  0.317 s]
[INFO] Apache Hadoop YARN ................................. SUCCESS [  0.291 s]
[INFO] Apache Hadoop YARN API ............................. SUCCESS [  8.560 s]
[INFO] Apache Hadoop YARN Common .......................... SUCCESS [ 29.751 s]
[INFO] Apache Hadoop YARN Registry ........................ SUCCESS [  4.885 s]
[INFO] Apache Hadoop YARN Server .......................... SUCCESS [  0.354 s]
[INFO] Apache Hadoop YARN Server Common ................... SUCCESS [  8.905 s]
[INFO] Apache Hadoop YARN NodeManager ..................... FAILURE [  2.680 s]
[INFO] Apache Hadoop YARN Web Proxy ....................... SKIPPED
[INFO] Apache Hadoop YARN ApplicationHistoryService ....... SKIPPED
[INFO] Apache Hadoop YARN Timeline Service ................ SKIPPED
[INFO] Apache Hadoop YARN ResourceManager ................. SKIPPED
[INFO] Apache Hadoop YARN Server Tests .................... SKIPPED
[INFO] Apache Hadoop YARN Client .......................... SKIPPED
[INFO] Apache Hadoop YARN SharedCacheManager .............. SKIPPED
[INFO] Apache Hadoop YARN Timeline Plugin Storage ......... SKIPPED
[INFO] Apache Hadoop YARN TimelineService HBase Backend ... SKIPPED
[INFO] Apache Hadoop YARN TimelineService HBase Common .... SKIPPED
[INFO] Apache Hadoop YARN TimelineService HBase Client .... SKIPPED
[INFO] Apache Hadoop YARN TimelineService HBase Servers ... SKIPPED
[INFO] Apache Hadoop YARN TimelineService HBase Server 1.2  SKIPPED
[INFO] Apache Hadoop YARN TimelineService HBase tests ..... SKIPPED
[INFO] Apache Hadoop YARN Router .......................... SKIPPED
[INFO] Apache Hadoop YARN Applications .................... SKIPPED
[INFO] Apache Hadoop YARN DistributedShell ................ SKIPPED
[INFO] Apache Hadoop YARN Unmanaged Am Launcher ........... SKIPPED
[INFO] Apache Hadoop MapReduce Client ..................... SKIPPED
[INFO] Apache Hadoop MapReduce Core ....................... SKIPPED
[INFO] Apache Hadoop MapReduce Common ..................... SKIPPED
[INFO] Apache Hadoop MapReduce Shuffle .................... SKIPPED
[INFO] Apache Hadoop MapReduce App ........................ SKIPPED
[INFO] Apache Hadoop MapReduce HistoryServer .............. SKIPPED
[INFO] Apache Hadoop MapReduce JobClient .................. SKIPPED
[INFO] Apache Hadoop Mini-Cluster ......................... SKIPPED
[INFO] Apache Hadoop YARN Services ........................ SKIPPED
[INFO] Apache Hadoop YARN Services Core ................... SKIPPED
[INFO] Apache Hadoop YARN Services API .................... SKIPPED
[INFO] Apache Hadoop YARN Site ............................ SKIPPED
[INFO] Apache Hadoop YARN UI .............................. SKIPPED
[INFO] Apache Hadoop YARN Project ......................... SKIPPED
[INFO] Apache Hadoop MapReduce HistoryServer Plugins ...... SKIPPED
[INFO] Apache Hadoop MapReduce NativeTask ................. SKIPPED
[INFO] Apache Hadoop MapReduce Uploader ................... SKIPPED
[INFO] Apache Hadoop MapReduce Examples ................... SKIPPED
[INFO] Apache Hadoop MapReduce ............................ SKIPPED
[INFO] Apache Hadoop MapReduce Streaming .................. SKIPPED
[INFO] Apache Hadoop Distributed Copy ..................... SKIPPED
[INFO] Apache Hadoop Archives ............................. SKIPPED
[INFO] Apache Hadoop Archive Logs ......................... SKIPPED
[INFO] Apache Hadoop Rumen ................................ SKIPPED
[INFO] Apache Hadoop Gridmix .............................. SKIPPED
[INFO] Apache Hadoop Data Join ............................ SKIPPED
[INFO] Apache Hadoop Extras ............................... SKIPPED
[INFO] Apache Hadoop Pipes ................................ SKIPPED
[INFO] Apache Hadoop OpenStack support .................... SKIPPED
[INFO] Apache Hadoop Amazon Web Services support .......... SKIPPED
[INFO] Apache Hadoop Kafka Library support ................ SKIPPED
[INFO] Apache Hadoop Azure support ........................ SKIPPED
[INFO] Apache Hadoop Aliyun OSS support ................... SKIPPED
[INFO] Apache Hadoop Client Aggregator .................... SKIPPED
[INFO] Apache Hadoop Scheduler Load Simulator ............. SKIPPED
[INFO] Apache Hadoop Resource Estimator Service ........... SKIPPED
[INFO] Apache Hadoop Azure Data Lake support .............. SKIPPED
[INFO] Apache Hadoop Image Generation Tool ................ SKIPPED
[INFO] Apache Hadoop Tools Dist ........................... SKIPPED
[INFO] Apache Hadoop Tools ................................ SKIPPED
[INFO] Apache Hadoop Client API ........................... SKIPPED
[INFO] Apache Hadoop Client Runtime ....................... SKIPPED
[INFO] Apache Hadoop Client Packaging Invariants .......... SKIPPED
[INFO] Apache Hadoop Client Test Minicluster .............. SKIPPED
[INFO] Apache Hadoop Client Packaging Invariants for Test . SKIPPED
[INFO] Apache Hadoop Client Packaging Integration Tests ... SKIPPED
[INFO] Apache Hadoop Distribution ......................... SKIPPED
[INFO] Apache Hadoop Client Modules ....................... SKIPPED
[INFO] Apache Hadoop Cloud Storage ........................ SKIPPED
[INFO] Apache Hadoop Cloud Storage Project ................ SKIPPED
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  03:35 min
[INFO] Finished at: 2019-02-21T15:31:33+08:00
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.apache.hadoop:hadoop-maven-plugins:3.1.1:cmake-compile (cmake-compile) on project hadoop-yarn-server-nodemanager: make failed with error code 2 -> [Help 1]
org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute goal org.apache.hadoop:hadoop-maven-plugins:3.1.1:cmake-compile (cmake-compile) on project hadoop-yarn-server-nodemanager: make failed with error code 2
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:215)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:156)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:148)
    at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:117)
    at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:81)
    at org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build (SingleThreadedBuilder.java:56)
    at org.apache.maven.lifecycle.internal.LifecycleStarter.execute (LifecycleStarter.java:128)
    at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:305)
    at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:192)
    at org.apache.maven.DefaultMaven.execute (DefaultMaven.java:105)
    at org.apache.maven.cli.MavenCli.execute (MavenCli.java:956)
    at org.apache.maven.cli.MavenCli.doMain (MavenCli.java:288)
    at org.apache.maven.cli.MavenCli.main (MavenCli.java:192)
    at sun.reflect.NativeMethodAccessorImpl.invoke0 (Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke (NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke (DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke (Method.java:498)
    at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced (Launcher.java:289)
    at org.codehaus.plexus.classworlds.launcher.Launcher.launch (Launcher.java:229)
    at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode (Launcher.java:415)
    at org.codehaus.plexus.classworlds.launcher.Launcher.main (Launcher.java:356)
Caused by: org.apache.maven.plugin.MojoExecutionException: make failed with error code 2
    at org.apache.hadoop.maven.plugin.cmakebuilder.CompileMojo.runMake (CompileMojo.java:229)
    at org.apache.hadoop.maven.plugin.cmakebuilder.CompileMojo.execute (CompileMojo.java:98)
    at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo (DefaultBuildPluginManager.java:137)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:210)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:156)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:148)
    at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:117)
    at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:81)
    at org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build (SingleThreadedBuilder.java:56)
    at org.apache.maven.lifecycle.internal.LifecycleStarter.execute (LifecycleStarter.java:128)
    at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:305)
    at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:192)
    at org.apache.maven.DefaultMaven.execute (DefaultMaven.java:105)
    at org.apache.maven.cli.MavenCli.execute (MavenCli.java:956)
    at org.apache.maven.cli.MavenCli.doMain (MavenCli.java:288)
    at org.apache.maven.cli.MavenCli.main (MavenCli.java:192)
    at sun.reflect.NativeMethodAccessorImpl.invoke0 (Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke (NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke (DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke (Method.java:498)
    at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced (Launcher.java:289)
    at org.codehaus.plexus.classworlds.launcher.Launcher.launch (Launcher.java:229)
    at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode (Launcher.java:415)
    at org.codehaus.plexus.classworlds.launcher.Launcher.main (Launcher.java:356)
[ERROR]
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException
[ERROR]
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR]   mvn <goals> -rf :hadoop-yarn-server-nodemanager
hadooppyarrow

1条答案

按热度按时间
u5rb5r59

u5rb5r591#

同样的问题，如果您仔细阅读编译日志，您将看到

[WARNING] /Users/wyx/Downloads/hadoop-rel-release-3.1.1/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/utils/docker-util.c:1227:12: error: no matching function for call to 'getgrouplist'
[WARNING]   int rc = getgrouplist(user, pw->pw_gid, groups, &ngroups);
[WARNING]            ^~~~~~~~~~~~
[WARNING] /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.14.sdk/usr/include/unistd.h:653:6: note: candidate function not viable: no known conversion from 'gid_t *' (aka 'unsigned int *') to 'int *' for 3rd argument
[WARNING] int      getgrouplist(const char *, int, int *, int *);

谷歌搜索发现了一个问题https://issues.apache.org/jira/browse/yarn-8622 在这里
下面的更改对我很有效：

source file: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/utils/docker-util.c
line 1227:
    -  int rc = getgrouplist(user, pw->pw_gid, groups, &ngroups);
    +  int rc = getgrouplist(user, pw->pw_gid, (int *)groups, &ngroups);
line 1234:
    -    if (getgrouplist(user, pw->pw_gid, groups, &ngroups) == -1) {
    +    if (getgrouplist(user, pw->pw_gid, (int *)groups, &ngroups) == -1) {
赞(0）分享  回复(0）举报  2021-05-27
我来回答

相关问题

    查看更多

    热门标签

    更多
    JavaquerypythonNode开发语言requestUtil数据库Table后端算法LoggerMessageElementParser

    最新问答

    更多
    • 蜀ICP备2022004421号-2 NULL123 https://www.null123.com © All rights reserved
    • 本站内容来源互联网,如果侵犯您的权益请联系我们删除, 联系方式：448109455@qq.com