如何使用Python将数据导入Azure群集

dsekswqp  于 2023-05-01  发布在  Python
关注(0)|答案(1)|浏览(93)

我有一组数据,我想使用Azure Data Explorer使用KQL查询。我每隔几秒就有一个连续的输入数据源。我想将这些数据放入Azure集群中运行查询。
我已经探索了一些使用python库的选项,但它只支持使用文件或blob。
https://learn.microsoft.com/en-us/azure/data-explorer/python-ingest-data
如何使用Python将单个记录放入,以便可以使用Azure Data Explorer进行查询。

vsnjm48y

vsnjm48y1#

这里有使用azure-kusto-ingest库从 Dataframe (具有一个或多个记录)中获取的示例代码

cluster = "https://ingest-{cluster_name}.kusto.windows.net/"

# In case you want to authenticate with AAD application.
client_id = "<insert here your AAD application id>"
client_secret = "<insert here your AAD application key>"

# read more at https://docs.microsoft.com/en-us/onedrive/find-your-office-365-tenant-id
authority_id = "<insert here your tenant id>"

kcsb = KustoConnectionStringBuilder.with_aad_application_key_authentication(cluster, client_id, client_secret, authority_id)

client = QueuedIngestClient(kcsb)

# there are a lot of useful properties, make sure to go over docs and check them out
ingestion_props = IngestionProperties(
    database="{database_name}",
    table="{table_name}",
    data_format=DataFormat.CSV,
    # in case status update for success are also required (remember to import ReportLevel from azure.kusto.ingest)
    # report_level=ReportLevel.FailuresAndSuccesses,
    # in case a mapping is required (remember to import IngestionMappingKind from azure.kusto.data.data_format)
    # ingestion_mapping_reference="{json_mapping_that_already_exists_on_table}",
    # ingestion_mapping_kind= IngestionMappingKind.JSON,
)

###########################
## ingest from dataframe ##
###########################

import pandas

fields = ["id", "name", "value"]
rows = [[1, "abc", 15.3], [2, "cde", 99.9]]

df = pandas.DataFrame(data=rows, columns=fields)

client.ingest_from_dataframe(df, ingestion_properties=ingestion_props)

相关问题