python PySpark从Colab中的Azure Blob存储读取-未找到org.apache.hadoop.fs.azure.NativeAzureFileSystem$Secure类

xe55xuns 于 2023-05-21 发布在 Python

关注(0)|答案(2)|浏览(163)

我一直试图在Jupyter notebooks/google colab中使用pyspark从Azure Blob Storage读取JSON数据，并不断遇到相同的错误-java.lang.RuntimeException: java.lang.ClassNotFoundException: Class org.apache.hadoop.fs.azure.NativeAzureFileSystem$Secure not found
下面的代码将是我如何在谷歌colab中设置的，但在Jupyter中的差异是最小的。
我的设置：
我使用的是以前版本的spark，因为我在最新版本3.4.0中遇到了这个问题

!apt-get install openjdk-8-jdk-headless -qq > /dev/null
!wget -q http://archive.apache.org/dist/spark/spark-3.1.1/spark-3.1.1-bin-hadoop3.2.tgz
!tar xf spark-3.1.1-bin-hadoop3.2.tgz
!pip install -q findspark

import os
os.environ["JAVA_HOME"] = "/usr/lib/jvm/java-8-openjdk-amd64"
os.environ["SPARK_HOME"] = "/content/spark-3.1.1-bin-hadoop3.2"

在spark目录-“/content/spark-3.1.1-bin-hadoop 3.2/jars”中-我放置了以下jar：

azure-storage-8.6.6.jar
hadoop-azure-3.5.5.jar
jetty-util-11.0.15.jar
jetty-util-ajax-11.0.15.jar

import findspark
findspark.init()
import pyspark
from pyspark.sql import SparkSession

conf = pyspark.SparkConf()
conf.set(
    "spark.jars.packages",
    "org.apache.hadoop:hadoop-azure-3.3.5,com.microsoft.azure:azure-storage-8.6.6"
    # I have also tried hadoop-azure:3.3.5 and azure-storage:8.6.6 (: instead of -)
)

conf.set(
    "fs.azure.account.key.<STORAGE_ACCOUNT>.blob.core.windows.net", "<TOKEN>")

spark = SparkSession.builder.master("local[*]")\
  .config(conf=conf)\
  .getOrCreate()
spark.conf.set("spark.sql.repl.eagerEval.enabled", True)
spark

然后尝试以几种不同的格式读取json和csv数据：

df1 = spark.read.format('json').\
    load("wasbs://{CONTAINER}@{ACCOUNT}.blob.core.windows.net/{FILE_PATH}.json")

df2 = spark.read.json("wasbs://{CONTAINER}@{ACCOUNT}.blob.core.windows.net/{FILE_PATH}.json")```

df3 = spark.read.csv("wasbs://{CONTAINER}@{ACCOUNT}.blob.core.windows.net/{FILE_PATH}.csv")```

上面的每一个都抛出错误：

Py4JJavaError: An error occurred while calling o233.load.
: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class org.apache.hadoop.fs.azure.NativeAzureFileSystem$Secure not found
    at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2595)
    at org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:3269)
    at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:3301)
    at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:124)
    at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:3352)
    at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:3320)
    at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:479)
    at org.apache.hadoop.fs.Path.getFileSystem(Path.java:361)
    at org.apache.spark.sql.execution.streaming.FileStreamSink$.hasMetadata(FileStreamSink.scala:46)
    at org.apache.spark.sql.execution.datasources.DataSource.resolveRelation(DataSource.scala:376)
    at org.apache.spark.sql.DataFrameReader.loadV1Source(DataFrameReader.scala:326)
    at org.apache.spark.sql.DataFrameReader.$anonfun$load$3(DataFrameReader.scala:308)
    at scala.Option.getOrElse(Option.scala:189)
    at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:308)
    at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:240)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)
    at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)
    at py4j.Gateway.invoke(Gateway.java:282)
    at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
    at py4j.commands.CallCommand.execute(CallCommand.java:79)
    at py4j.GatewayConnection.run(GatewayConnection.java:238)
    at java.lang.Thread.run(Thread.java:750)
Caused by: java.lang.ClassNotFoundException: Class org.apache.hadoop.fs.azure.NativeAzureFileSystem$Secure not found
    at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:2499)
    at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2593)
    ... 25 more

我已确保我的Azure帐户信息，密钥和文件路径正确，并且通常读取pyspark的工作-通过以下方式验证：

df = spark.read.csv('sample_data/california_housing_test.csv', header=True, sep=";")
df.show(5)

我已经按照指示，从here和here和here试图解决我的问题，但无济于事，请告知！

python

来源：https://stackoverflow.com/questions/76256394/pyspark-read-from-azure-blob-storage-in-colab-class-org-apache-hadoop-fs-azure

2条答案

按热度按时间

vulvrdjw1#

我尝试使用以下代码从Google Collab中的Azure Blob存储中读取数据：

from pyspark.sql import SparkSession

spark = SparkSession.builder \  
.master("spark://spark-master-svc:7077") \  
.config("spark.hadoop.fs.azure.account.key.{account_name}.blob.core.windows.net", account_key) \
.getOrCreate()
account_name = '<storageaccountName>'

account_key = '<accesskey>'
container_name = '<containerName>'
df = spark.read.json(f"wasbs://{container_name}@{account_name}.blob.core.windows.net/input.json")

我得到了以下错误：

我尝试了下面的代码，我能够从blob存储中读取数据：

from datetime import datetime, timedelta
from azure.storage.blob import BlobServiceClient, generate_blob_sas, BlobSasPermissions
import pandas as pd

account_name = '<storageaccount>'
account_key = '<AccessKey>'
container_name = '<containerName>'

connect_str = 'DefaultEndpointsProtocol=https;AccountName=' + account_name + ';AccountKey=' + account_key + ';EndpointSuffix=core.windows.net'
blob_service_client = BlobServiceClient.from_connection_string(connect_str)
container_client = blob_service_client.get_container_client(container_name)

blob_list = []
for blob_i in container_client.list_blobs():
     blob_list.append(blob_i.name)
     
for blob_i in blob_list:
  sas_i = generate_blob_sas(account_name = account_name,
    container_name = container_name,
    blob_name = blob_i,
    account_key=account_key,
    permission=BlobSasPermissions(read=True),
    expiry=datetime.utcnow() + timedelta(hours=1))

  sas_url = 'https://' + account_name+'.blob.core.windows.net/' + container_name + '/' + blob_i + '?' + sas_i
  df = pd.read_csv(sas_url)
  print(df)

赞(0）回复(0）举报 2023-05-21

im9ewurl2#

在尝试了配置设置以及如何将jar带入环境之后，我能够让它工作：

!curl https://repo1.maven.org/maven2/org/apache/hadoop/hadoop-azure/3.3.5/hadoop-azure-3.3.5.jar --output /content/spark-3.1.1-bin-hadoop3.2/jars/hadoop-azure-3.3.5.jar
!curl https://repo1.maven.org/maven2/com/microsoft/azure/azure-storage/8.6.6/azure-storage-8.6.6.jar --output /content/spark-3.1.1-bin-hadoop3.2/jars/azure-storage-8.6.6.jar

注意“spark.jars.packages”更改为“spark.jars”，然后指向上面定义的文件路径。

conf = pyspark.SparkConf()
conf.set(
    "spark.jars",
    "/content/spark-3.1.1-bin-hadoop3.2/jars/hadoop-azure-3.3.5.jar, /content/spark-3.1.1-bin-hadoop3.2/jars/azure-storage-8.6.6.jar"
)

**请注意，此修复在Colab中有效，我仍然没有完全让Jupyter notebook正常工作。

赞(0）回复(0）举报 2023-05-21

我来回答

python PySpark从Colab中的Azure Blob存储读取-未找到org.apache.hadoop.fs.azure.NativeAzureFileSystem$Secure类

2条答案

相关问题

热门标签

最新问答