Apache Spark 未找到容器,无法在Azure数据块中使用匿名凭据创建容器

3z6pesqy  于 2023-04-07  发布在  Apache
关注(0)|答案(1)|浏览(133)

我有下面的配置,我从databricks阅读文件从我的blob存储.我的第一个配置工作,它使用SAS密钥从blob容器.和下一个是一个我无法读取与存储帐户密钥.我得到的错误是
未找到account . www.example.com中的容器blob.core.windows.net,无法使用匿名凭据创建该容器,
工作的配置:

spark.conf.set("fs.azure.sas.containername.storageaccount.blob.core.windows.net","sas-key")
dbutils.fs.ls("wasbs://containername@storageaccount.blob.core.windows.net/")
table_name= "main.deltalake_db.Customers_Log_Table_test"
checkpoint_path = "wasbs://testaudit@azdevstoreforlogs.blob.core.windows.net/_checkpoint"
file_path = "wasbs://containername@storageaccount.blob.core.windows.net/topics/logs/"
schema = "xHeaderFields ARRAY<STRUCT<key: STRING, value: STRING>>"
(spark.readStream
  .format("cloudFiles")
  .option("cloudFiles.format", "json")
  .option("cloudFiles.schemaLocation", checkpoint_path)
  .schema(schema)
  .load(file_path)
  .writeStream
  .option("checkpointLocation", checkpoint_path)
  .trigger(availableNow=True)
  .toTable(table_name))

不工作的是:(我得到以上错误)

dbutils.fs.mount(
source = "wasbs://containername@storageaccount.blob.core.windows.net",
mount_point = "/mnt/topics/logs",
extra_configs = {"fs.azure.account.key.storageaccount.blob.core.windows.net":dbutils.secrets.get(scope="keyvaultscope",key="storage-account-azuredevauditlogs")})
table_name= "main.deltalake_db.Customers_Log_Table_test"
checkpoint_path = "wasbs://testaudit@azdevstoreforlogs.blob.core.windows.net/_checkpoint"
file_path = "wasbs://containername@storageaccount.blob.core.windows.net/topics/logs/"
schema = "xHeaderFields ARRAY<STRUCT<key: STRING, value: STRING>>"
(spark.readStream
  .format("cloudFiles")
  .option("cloudFiles.format", "json")
  .option("cloudFiles.schemaLocation", checkpoint_path)
  .schema(schema)
  .load(file_path)
  .writeStream
  .option("checkpointLocation", checkpoint_path)
  .trigger(availableNow=True)
  .toTable(table_name))
yhuiod9q

yhuiod9q1#

找出答案..你需要在文件路径中有相同的挂载点。在上面的代码中,将file_path替换为下面的:

file_path ="/mnt/topics/logs"

其与安装路径中提供的相同

相关问题