目标

我正在用databricksconnect（v6.6）将spark应用程序从本地机器（客户机模式）提交到databricks集群。如Spark测量页面所述，使用pypi sparkmeasure==0.14.0 .

问题

为什么spark measure不打印任何指标？是否可以使用Spark测量与数据桥连接？

代码

spark = SparkSession \
    .builder \
    .appName(app_name) \
    .config("spark.jars.packages", "ch.cern.sparkmeasure:spark-measure_2.11:0.16") \
    .config("spark.driver.host", "localhost") \
    .config("spark.driver.bindAddress", "127.0.0.1") \
    .config("fs.azure.account.key.<my_storage>.dfs.core.windows.net", key) \
    .getOrCreate()

 from sparkmeasure import StageMetrics, TaskMetrics
 df = load_data(some_path)
 StageMetrics(self.spark).runandmeasure(locals(), 'df.count()'). # output 1

 df2 = load_data(some_path)
 TaskMetrics(self.spark).runandmeasure(locals(), 'df2.count()'). # output 2

输出1

Scheduling mode = FIFO
Spark Context default degree of parallelism = 8
 no data to report

输出2

Scheduling mode = FIFO
Spark Contex default degree of parallelism = 8
Aggregated Spark task metrics:
numtasks => 0
elapsedTime => null
sum(duration) => null
sum(schedulerDelay) => null
sum(executorRunTime) => null
sum(executorCpuTime) => null
sum(executorDeserializeTime) => null
sum(executorDeserializeCpuTime) => null
sum(resultSerializationTime) => null
sum(jvmGCTime) => null
sum(shuffleFetchWaitTime) => null
sum(shuffleWriteTime) => null
sum(gettingResultTime) => null
max(resultSize) => null
sum(numUpdatedBlockStatuses) => null
sum(diskBytesSpilled) => null
sum(memoryBytesSpilled) => null
max(peakExecutionMemory) => null
sum(recordsRead) => null
sum(bytesRead) => null
sum(recordsWritten) => null
sum(bytesWritten) => null
sum(shuffleTotalBytesRead) => null
sum(shuffleTotalBlocksFetched) => null
sum(shuffleLocalBlocksFetched) => null
sum(shuffleRemoteBlocksFetched) => null
sum(shuffleBytesWritten) => null
sum(shuffleRecordsWritten) => null