pyspark 在Spark中使用java_method从Java数组中获取值

yhuiod9q  于 2023-06-21  发布在  Spark
关注(0)|答案(1)|浏览(182)

使用Spark的java_method(以及来自java.util.LocalegetISOCountries),我尝试访问所有国家的列表。我没有得到任何错误,但返回的值看起来像[Ljava.lang.String;@5a68a908。当我使用类似的方法获得货币时,我得到的是正确的。

from pyspark.sql import functions as F
df = spark.range(1).select(
    F.expr("java_method('java.util.Locale', 'getISOCountries')").alias('countries'),
    F.expr("java_method('java.util.Currency', 'getAvailableCurrencies')").alias('currencies'),
)
df.show(truncate=0)
# +----------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
# |countries                   |currencies                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    |
# +----------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
# |[Ljava.lang.String;@5a68a908|[XPF, SDD, KZT, MXN, THB, XCD, KPW, BOB, DZD, PGK, CLF, BYN, LKR, BWP, IQD, UYU, HUF, PEN, SGD, HRK, XAU, MVR, SAR, UYI, ZWN, LVL, NLG, ZAR, ZWD, SLL, PLN, KGS, BIF, RUR, CHE, TRL, TMT, TZS, BTN, AFN, SRG, GIP, KMF, HTG, AYM, JMD, USS, RSD, SEK, XAF, XBD, UGX, YER, GRD, FKP, CAD, HKD, ILS, ADP, TJS, JPY, LUF, ZWL, DEM, ITL, BAM, BYR, ZMW, BRL, SRD, INR, STD, ETB, NPR, GHS, NZD, AWG, RON, RWF, XTS, NGN, COU, NIO, VEB, BEF, BYB, TRY, BGN, MUR, VUV, SYP, XPT, DOP, MRU, XBC, ISK, LTL, BMD, CSD, SZL, CRC, TWD, SDG, GHC, MZN, FIM, ESP, MGA, ALL, USN, XDR, ZMK, XBB, XPD, MTL, SOS, AZN, MMK, SLE, PHP, KWD, XOF, AED, MDL, CHF, CUP, BDT, JOD, BSD, CHW, HNL, XXX, STN, AZM, ANG, BND, VES, SSP, PYG, EEK, NOK, MWK, BZD, SBD, MAD, ROL, TOP, ZWR, VED, EUR, LAK, TPE, CUC, GBP, SHP, KHR, IEP, TTD, CZK, XBA, QAR, BBD, MXV, TND, VEF, OMR, KES, LSL, VND, PTE, XAG, KRW, LRD, GMD, UAH, IRR, SCR, UZS, XUA, FJD, BGL, CNY, DKK, TMM, CLP, ARS, PAB, RUB, GNF, SVC, GTQ, CYP, EGP, KYD, PKR, DJF, FRF, AFA, USD, CVE, MNT, LBP, XSU, SIT, NAD, BOV, GWP, MZM, ATS, MGF, IDR, MKD, WST, AOA, XFO, COP, MRO, XFU, YUM, GYD, MOP, GEL, SKK, AMD, LYD, MYR, ERN, AUD, BHD, CDF]|
# +----------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

这可能是因为getISOCountries返回一个数组,而getAvailableCurrencies返回一个集合。
有没有一种方法可以“解析”返回的getISOCountries数组/列表值,这样我就可以得到一个正确的国家列表?

nhhxz33t

nhhxz33t1#

如果不使用java_method,实现此目的的一种方法如下所示:

>>> x = sc._jvm.java.util.Locale.getISOCountries()
>>> xx = ', '.join(x)
>>> spark.range(1).withColumn('countries',lit(xx)).show()
+---+--------------------+
| id|           countries|
+---+--------------------+
|  0|AD, AE, AF, AG, A...|
+---+--------------------+

值得注意的是,“国家”(以及原始df中的“货币”)是一个StringType列,而不是ArrayType

相关问题