从pyspark的列中删除非英语单词

qmelpv7a 于 2021-07-13 发布在 Spark

关注(0)|答案(1)|浏览(448)

我正在处理pysparkDataframe，如下所示：

+-------+--------------------------------------------------+
|     id|                                             words|
+-------+--------------------------------------------------+
|1475569|[pt, m, reporting, delivery, scam, thank, 0a, 0...|
|1475568|[, , delivered, trblake, yahoo, com, received, ...|
|1475566|[,  marco, v, washin, gton, thursday, de, cembe...|
|1475565|[, marco, v, washin, gton, wednesday, de, cembe...|
|1475563|[joyce, 20, begin, forwarded, message, 20, memo...|
+-------+--------------------------------------------------+

数据类型：

id: 'bigint'
words: 'array<string>'

我想从“单词”列中删除非英语单词（包括数值或带数字的单词，如bun20），我已经删除了停止词，但如何从该列中删除其他非英语单词？
请帮忙。

python apache-spark pyspark data-cleaning non-english

来源：https://stackoverflow.com/questions/66367953/remove-non-english-words-from-column-in-pyspark

1条答案

按热度按时间

h9a6wy2h1#

您可以使用自定义项检查数组中的每个单词是否在nltk语料库中：

import pyspark.sql.functions as F
import nltk
from nltk.stem import WordNetLemmatizer
wnl = WordNetLemmatizer()

nltk.download('words')
nltk.download('wordnet')

@F.udf('array<string>')
def remove_words(words):
    return [word for word in words if wnl.lemmatize(word) in nltk.corpus.words.words()]

df2 = df.withColumn('words', remove_words('words'))

赞(0）回复(0）举报 2021-07-13

我来回答

从pyspark的列中删除非英语单词

1条答案

相关问题

热门标签

最新问答