hadoop压缩错误:java.lang.unsatifiedLinkError:org.apache.hadoop.util.nativecodeloader.buildsupportssnappy()z

lsmd5eda  于 2021-05-27  发布在  Hadoop
关注(0)|答案(0)|浏览(849)

在hadoop上运行apache kylin时,遇到了以下与hadoop mapreduce相关的错误:

2019-03-20 08:06:00,193 ERROR [main] org.apache.kylin.engine.mr.KylinMapper: java.lang.UnsatisfiedLinkError: org.apache.hadoop.util.NativeCodeLoader.buildSupportsSnappy()Z     
at org.apache.hadoop.util.NativeCodeLoader.buildSupportsSnappy(Native Method)     
at org.apache.hadoop.io.compress.SnappyCodec.checkNativeCodeLoaded(SnappyCodec.java:63)     
at org.apache.hadoop.io.compress.SnappyCodec.getCompressorType(SnappyCodec.java:136)     
at org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:150)     
at org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:168)     
at org.apache.hadoop.io.SequenceFile$Writer.init(SequenceFile.java:1304)     
at org.apache.hadoop.io.SequenceFile$Writer.<init>(SequenceFile.java:1192)     
at org.apache.hadoop.io.SequenceFile$BlockCompressWriter.<init>(SequenceFile.java:1552)     
at org.apache.hadoop.io.SequenceFile.createWriter(SequenceFile.java:289)     
at org.apache.hadoop.io.SequenceFile.createWriter(SequenceFile.java:542)     
at org.apache.hadoop.mapreduce.lib.output.SequenceFileOutputFormat.getSequenceWriter(SequenceFileOutputFormat.java:64) 
at org.apache.hadoop.mapreduce.lib.output.SequenceFileOutputFormat.getRecordWriter(SequenceFileOutputFormat.java:75)     
at org.apache.hadoop.mapreduce.lib.output.LazyOutputFormat$LazyRecordWriter.write(LazyOutputFormat.java:113)     
at org.apache.hadoop.mapreduce.lib.output.MultipleOutputs.write(MultipleOutputs.java:468)

我想原因是hadoop找不到libsnapy.so*本机库。我已经在网上搜索了解决方案。通过此链接,我已经在相应的xml文件中添加了以下属性,并重新启动了服务:


# For HDFS core-site.xml

<property>
  <name>io.compression.codecs</name>
  <value>
    org.apache.hadoop.io.compress.GzipCodec,org.apache.hadoop.io.compress.DefaultCodec,org.apache.hadoop.io.compress.SnappyCodec
  </value>
</property>

# For MapReduce2 mapred-site.xml

<property>
  <name>mapreduce.admin.user.env</name>
  <value>LD_LIBRARY_PATH=/usr/hdp/${hdp.version}/hadoop/lib/native</value>
</propert
<property> 
  <name>mapreduce.map.output.compress</name>  
  <value>true</value>
</property>
<property>
  <name>mapred.map.output.compress.codec</name>  
  <value>org.apache.hadoop.io.compress.SnappyCodec</value>
</property>
<property>
  <name>mapreduce.output.fileoutputformat.compress</name>
  <value>true</value> 
</property> 
<property> 
  <name>mapreduce.output.fileoutputformat.compress.codec</name>
  <value>org.apache.hadoop.io.compress.SnappyCodec</value> 
</property>

然而,它没有起作用。所以我就在原木上挖Yarn。我发现在launch\u container.sh部分中,它有以下命令:

export PWD="/hadoop/yarn/local/usercache/root/appcache/application_1553049994285_0013/container_e04_1553049994285_0013_01_000005"

# ...omit other commands

export LD_LIBRARY_PATH="$PWD"

我认为这个命令是错误的,因为到libsnapy.so*的真正路径不是$pwd。

LD_LIBRARY_PATH=/usr/hdp/${hdp.version}/hadoop/lib/native

同样如您所见,我已经将ld\u library\u path设置为指向mapred-site.xml文件中的真实路径。为什么Yarn仍然使用$pwd?
此外,我还添加了一条日志消息,如下所示。这就确保了env ld\u library\u路径确实设置错误。那我怎么解决这个问题呢?

2019-03-20 08:06:00,044 INFO [main] org.apache.kylin.engine.mr.KylinMapper: linyanwen[from map]: /hadoop/yarn/local/usercache/root/appcache/application_1553049994285_0039/container_e04_1553049994285_0039_01_000005

谢谢!

暂无答案!

目前还没有任何答案,快来回答吧!

相关问题