pyspark 使用Spark dataframe计算字符串列中的子字符串

f45qwnt8  于 2023-06-28  发布在  Spark
关注(0)|答案(4)|浏览(191)

我有一个Spark Dataframe ,其中包含字符串类型的列(assigned_products),其中包含以下值:

"POWER BI PRO+Power BI (free)+AUDIO CONFERENCING+OFFICE 365 ENTERPRISE E5 WITHOUT AUDIO CONFERENCING"

我想计算+在字符串中出现的次数,并在新列中返回该值。
我尝试了以下方法,但总是返回错误。

from pyspark.sql.functions import col

DF.withColumn('Number_Products_Assigned', col("assigned_products").count("+"))

我在运行Apache Spark 2.3.1的群集上的Azure Databricks中运行代码。

sz81bmfz

sz81bmfz1#

这里有一个非udf解决方案。将字符串分割为你要计数的字符,你想要的值是结果数组的长度减去1:

from pyspark.sql.functions import col, size, split
DF.withColumn('Number_Products_Assigned', size(split(col("assigned_products"), r"\+")) - 1)

你必须转义+,因为它是一个特殊的正则表达式字符。

+--------------------+------------------------+
|   assigned_products|Number_Products_Assigned|
+--------------------+------------------------+
|POWER BI PRO+Powe...|                       3|
+--------------------+------------------------+
goqiplq2

goqiplq22#

from pyspark.sql.functions import col,udf

@udf(returnType='int')
def cnt(s):
    return s.count('+')

DF.withColumn('Number_Products_Assigned', cnt(col("assigned_products")))
nmpmafwu

nmpmafwu3#

Replace将子字符串的出现替换为空字符串。因此,我们可以通过比较替换前后的长度来计算出现次数,如下所示:

使用SparkSQL:

SELECT length(x) - length(replace(x,'+')) as substring_count
FROM  (select 'abc+def+ghi++aaa' as x) -- Sample data

输出:

substring_count
---------------
4

使用PySpark函数:

import pyspark.sql.functions as F

df1 = spark.sql("select 'abc+def+ghi++aaa' as x") # Sample data
df1.withColumn('substring_count', 
                F.length(col('x')) 
               - F.length(F.regexp_replace(col('x'), '\+', '')) 
              ).show()

输出:

+----------------+---------------+
|               x|substring_count|
+----------------+---------------+
|abc+def+ghi++aaa|              4|
+----------------+---------------+
3b6akqbq

3b6akqbq4#

**Spark 3.4+**有regexp_count

F.expr(r"regexp_count(col_name, '\\+')")

完整示例:

from pyspark.sql import functions as F
df = spark.createDataFrame(
    [("POWER BI PRO+Power BI (free)+AUDIO CONFERENCING+OFFICE 365 ENTERPRISE E5 WITHOUT AUDIO CONFERENCING",)],
    ["assigned_products"])

df = df.withColumn('Number_Products_Assigned', F.expr(r"regexp_count(assigned_products, '\\+')"))

df.show()
# +--------------------+------------------------+
# |   assigned_products|Number_Products_Assigned|
# +--------------------+------------------------+
# |POWER BI PRO+Powe...|                       3|
# +--------------------+------------------------+

相关问题