PySpark删除所有特殊字符的所有列名中的特殊字符

kuarbcqp 于 12个月前发布在 Apache

关注(0)|答案(4)|浏览(130)

我正在尝试删除所有列中的所有特殊字符。我使用以下命令：

import pyspark.sql.functions as F

df_spark = spark_df.select([F.col(col).alias(col.replace(' ', '_')) for col in df.columns])
df_spark1 = df_spark.select([F.col(col).alias(col.replace('%', '_')) for col in df_spark.columns])
df_spark = df_spark1.select([F.col(col).alias(col.replace(',', '_')) for col in df_spark1.columns])
df_spark1 = df_spark.select([F.col(col).alias(col.replace('(', '_')) for col in df_spark.columns])
df_spark2 = df_spark1.select([F.col(col).alias(col.replace(')', '_')) for col in df_spark1.columns])

字符串
有没有更简单的方法可以在一个命令中替换所有特殊字符（不仅仅是上面的5个）？我在Databricks上使用PySpark。

apache-spark

来源：https://stackoverflow.com/questions/62441435/pyspark-remove-special-characters-in-all-column-names-for-all-special-characters

4条答案

按热度按时间

h9vpoimq1#

您可以替换除A-z和0-9以外的任何字符

import pyspark.sql.functions as F
import re

df = df.select([F.col(column_name).alias(re.sub("[^0-9a-zA-Z$]+", "", column_name)) for column_name in df.columns])

字符串

赞(0）回复(0）举报 12个月前

vfh0ocws2#

在python中使用re（regex）模块和list comprehension。

Example:

df=spark.createDataFrame([('a b','ac','ac','ac','ab')],["i d","id,","i(d","i)k","i%j"])

df.columns
#['i d', 'id,', 'i(d', 'i)k', 'i%j']

import re

#replacing all the special characters using list comprehension
[re.sub('[\)|\(|\s|,|%]','',x) for x in df.columns]
#['id', 'id', 'id', 'ik', 'ij']

df.toDF(*[re.sub('[\)|\(|\s|,|%]','',x) for x in df.columns])
#DataFrame[id: string, id: string, id: string, ik: string, ij: string]

字符串

赞(0）回复(0）举报 12个月前

gwbalxhn3#

**re.sub('[^\w]', '_', c)**将标点符号和空格替换为_下划线。

测试结果：

from pyspark.sql import SparkSession
import re

spark = SparkSession.builder.getOrCreate()
df = spark.createDataFrame([(1, 2, 3, 4)], [' 1', '%2', ',3', '(4)'])

df = df.toDF(*[re.sub('[^\w]', '_', c) for c in df.columns])
df.show()

#  +---+---+---+---+
#  | _1| _2| _3|_4_|
#  +---+---+---+---+
#  |  1|  2|  3|  4|
#  +---+---+---+---+

字符串
删除标点符号+使用_代替空格：
re.sub('[^\w ]', '', c).replace(' ', '_')个

赞(0）回复(0）举报 12个月前

bybem2ql4#

也许这是有用的-

// [^0-9a-zA-Z]+ => this will remove all special chars
    spark.range(2).withColumn("str", lit("abc%xyz_12$q"))
      .withColumn("replace", regexp_replace($"str", "[^0-9a-zA-Z]+", "_"))
      .show(false)

    /**
      * +---+------------+------------+
      * |id |str         |replace     |
      * +---+------------+------------+
      * |0  |abc%xyz_12$q|abc_xyz_12_q|
      * |1  |abc%xyz_12$q|abc_xyz_12_q|
      * +---+------------+------------+
      */

    // if you don't want to remove some special char like $ etc, include it [^0-9a-zA-Z$]+
    spark.range(2).withColumn("str", lit("abc%xyz_12$q"))
      .withColumn("replace", regexp_replace($"str", "[^0-9a-zA-Z$]+", "_"))
      .show(false)

    /**
      * +---+------------+------------+
      * |id |str         |replace     |
      * +---+------------+------------+
      * |0  |abc%xyz_12$q|abc_xyz_12$q|
      * |1  |abc%xyz_12$q|abc_xyz_12$q|
      * +---+------------+------------+
      */

字符串

赞(0）回复(0）举报 12个月前

我来回答

PySpark删除所有特殊字符的所有列名中的特殊字符

4条答案

相关问题

热门标签

最新问答