我有下面的配置,我从databricks阅读文件从我的blob存储.我的第一个配置工作,它使用SAS密钥从blob容器.和下一个是一个我无法读取与存储帐户密钥.我得到的错误是
未找到account . www.example.com中的容器blob.core.windows.net,无法使用匿名凭据创建该容器,
工作的配置:
spark.conf.set("fs.azure.sas.containername.storageaccount.blob.core.windows.net","sas-key")
dbutils.fs.ls("wasbs://containername@storageaccount.blob.core.windows.net/")
table_name= "main.deltalake_db.Customers_Log_Table_test"
checkpoint_path = "wasbs://testaudit@azdevstoreforlogs.blob.core.windows.net/_checkpoint"
file_path = "wasbs://containername@storageaccount.blob.core.windows.net/topics/logs/"
schema = "xHeaderFields ARRAY<STRUCT<key: STRING, value: STRING>>"
(spark.readStream
.format("cloudFiles")
.option("cloudFiles.format", "json")
.option("cloudFiles.schemaLocation", checkpoint_path)
.schema(schema)
.load(file_path)
.writeStream
.option("checkpointLocation", checkpoint_path)
.trigger(availableNow=True)
.toTable(table_name))
不工作的是:(我得到以上错误)
dbutils.fs.mount(
source = "wasbs://containername@storageaccount.blob.core.windows.net",
mount_point = "/mnt/topics/logs",
extra_configs = {"fs.azure.account.key.storageaccount.blob.core.windows.net":dbutils.secrets.get(scope="keyvaultscope",key="storage-account-azuredevauditlogs")})
table_name= "main.deltalake_db.Customers_Log_Table_test"
checkpoint_path = "wasbs://testaudit@azdevstoreforlogs.blob.core.windows.net/_checkpoint"
file_path = "wasbs://containername@storageaccount.blob.core.windows.net/topics/logs/"
schema = "xHeaderFields ARRAY<STRUCT<key: STRING, value: STRING>>"
(spark.readStream
.format("cloudFiles")
.option("cloudFiles.format", "json")
.option("cloudFiles.schemaLocation", checkpoint_path)
.schema(schema)
.load(file_path)
.writeStream
.option("checkpointLocation", checkpoint_path)
.trigger(availableNow=True)
.toTable(table_name))
1条答案
按热度按时间yhuiod9q1#
找出答案..你需要在文件路径中有相同的挂载点。在上面的代码中,将file_path替换为下面的:
其与安装路径中提供的相同