计算pyspark Dataframe 中的地理距离

68de4m5k 于 2023-02-07 发布在 Spark

关注(0)|答案(1)|浏览(201)

我的 Dataframe ：

DF = spark.createDataFrame([[114.038696,  22.5315,  114.047302, 22.531799], [ 114.027901, 22.5228, 114.026299, 22.5238], [ 114.026299, 22.5238,114.024597,22.5271], [114.024597,  22.5271,114.024696,22.527201]], list('ABCD'))
DF.show()
+----------+-------+----------+---------+
|         A|      B|         C|        D|
+----------+-------+----------+---------+
|114.038696|22.5315|114.047302|22.531799|
|114.027901|22.5228|114.026299|  22.5238|
|114.026299|22.5238|114.024597|  22.5271|
|114.024597|22.5271|114.024696|22.527201|
+----------+-------+----------+---------+

(A, B)&(C, D)为两点坐标;
column A和column C为纬度;
column B和column D为经度;
我想计算这两点之间的地理距离。
我试着：

from geopy.distance import geodesic
DF = DF.withColumn('Lengths/m', geodesic((['B'],['A']), (['D'],['C'])).m)

然后我得到错误：
TypeError：float（）参数必须是字符串或数字，而不是"list"
要成功计算地理距离，我应该采取哪些不同的做法？

pyspark

来源：https://stackoverflow.com/questions/70315069/calculate-the-geographical-distance-in-pyspark-dataframe

1条答案

按热度按时间

fkaflof61#

您需要定义一个自定义user-defined-function：

from geopy.distance import geodesic
import pyspark.sql.functions as F

@F.udf(returnType=FloatType())
def geodesic_udf(a, b):
    return geodesic(a, b).m

DF = DF.withColumn('Lengths/m', geodesic_udf(F.array("B", "A"), F.array("D", "C")))

DF.show()
#+----------+-------+----------+---------+---------+
#|A         |B      |C         |D        |Lengths/m|
#+----------+-------+----------+---------+---------+
#|114.038696|22.5315|114.047302|22.531799|885.94244|
#|114.027901|22.5228|114.026299|22.5238  |198.55937|
#|114.026299|22.5238|114.024597|22.5271  |405.21692|
#|114.024597|22.5271|114.024696|22.527201|15.126849|
#+----------+-------+----------+---------+---------+

赞(0）回复(0）举报 2023-02-07

我来回答

计算pyspark Dataframe 中的地理距离

1条答案

相关问题

热门标签

最新问答