我在emr上使用一个主节点和一个核心节点的spark,我面临以下问题:
java.lang.RuntimeException: native snappy library not available: this version of libhadoop was built without snappy support.
我将java.library.path设置为/usr/lib/hadoop/lib/native/:/usr/lib/hadoop lzo/lib/native/
checknative在主节点和核心节点上提供了以下信息:
16/04/28 15:03:27 INFO bzip2.Bzip2Factory: Successfully loaded & initialized native-bzip2 library system-native
16/04/28 15:03:27 INFO zlib.ZlibFactory: Successfully loaded & initialized native-zlib library
Native library checking:
hadoop: true /usr/lib/hadoop/lib/native/libhadoop.so.1.0.0
zlib: true /lib64/libz.so.1
snappy: true /usr/lib/hadoop/lib/native/libsnappy.so.1
lz4: true revision:99
bzip2: true /lib64/libbz2.so.1
openssl: true /usr/lib64/libcrypto.so
emr版本:emr-4.6.0
hadoop磁盘tribution:amazon 2.7.2
应用程序lications:spark 1.6.1
1条答案
按热度按时间qlvxas9a1#
我终于找到了解决方案,我设置了ld\u library\u路径,现在就可以工作了!!
我在所有节点上都做了这个操作。我希望它能帮助别人:)!