osx get pyarrow.lib.arrowioerror:无法加载libhdfs

c90pui9n  于 2021-05-27  发布在  Hadoop
关注(0)|答案(0)|浏览(346)
import pyarrow as pa

client = pa.hdfs.connect('localhost', 9000)

错误

Traceback (most recent call last):
  File "/Users/wyx/project/py3.7aio/hdfs/list_dir.py", line 13, in <module>
    client = pa.hdfs.connect('localhost', 9000)
  File "/Users/wyx/project/py3.7aio/.env/lib/python3.6/site-packages/pyarrow/hdfs.py", line 207, in connect
    extra_conf=extra_conf)
  File "/Users/wyx/project/py3.7aio/.env/lib/python3.6/site-packages/pyarrow/hdfs.py", line 38, in __init__
    self._connect(host, port, user, kerb_ticket, driver, extra_conf)
  File "pyarrow/io-hdfs.pxi", line 89, in pyarrow.lib.HadoopFileSystem._connect
  File "pyarrow/error.pxi", line 83, in pyarrow.lib.check_status
pyarrow.lib.ArrowIOError: Unable to load libhdfs

我安装hadoop brew 获取任何本机lib,因此我按照本机库指南构建hadoop3.1.1,但是我无法获取任何libhdfs.pyarrow需要的是libhdfs.dylib

➜  native git:(branch-3.1.1) ✗ hadoop checknative -a
2019-02-24 22:05:31,686 WARN bzip2.Bzip2Factory: Failed to load/initialize native-bzip2 library system-native, will use pure-Java version
2019-02-24 22:05:31,689 INFO zlib.ZlibFactory: Successfully loaded & initialized native-zlib library
2019-02-24 22:05:31,695 WARN erasurecode.ErasureCodeNative: ISA-L support is not available in your platform... using builtin-java codec where applicable
Native library checking:
hadoop:  true /usr/local/Cellar/hadoop/3.1.1/libexec/lib/native/libhadoop.dylib
zlib:    true /usr/lib/libz.1.dylib
zstd  :  false
snappy:  true /usr/local/lib/libsnappy.1.dylib
lz4:     true revision:10301
bzip2:   false
openssl: false build does not support openssl.
ISA-L:   false libhadoop was built without ISA-L support
2019-02-24 22:05:31,723 INFO util.ExitUtil: Exiting with status 1: ExitException

暂无答案!

目前还没有任何答案,快来回答吧!

相关问题