pandas 计算字符串之间的余弦相似度时未获得预期结果

yqlxgs2m 于 2023-03-06 发布在其他

关注(0)|答案(1)|浏览(116)

我想计算Pandas数据框中同一行的两个字符串之间的成对余弦相似度。
我使用了以下代码行：

import pandas as pd
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.feature_extraction.text import CountVectorizer
from sklearn.metrics.pairwise import cosine_similarity

pd.set_option('display.float_format', '{:.4f}'.format)

df = pd.DataFrame({'text1': ['The quick brown fox jumps over the lazy dog', 'The red apple', 'The big blue sky'],
                   'text2': ['The lazy cat jumps over the brown dog', 'The red apple', 'The big yellow sun']})

vectorizer = CountVectorizer().fit_transform(df['text1'] + ' ' + df['text2'])

cosine_similarities = cosine_similarity(vectorizer)[:, 0:1]

df['cosine_similarity'] = cosine_similarities

print(df)

它给了我以下输出，这似乎是不正确的：

有人能帮我找出我做错了什么吗？
谢谢你。

pandas

来源：https://stackoverflow.com/questions/75621922/didnt-get-the-expected-results-when-calculate-cosine-similarity-between-strings

1条答案

按热度按时间

q0qdq0h21#

我不是Maven，但有个办法。

import pandas as pd
import numpy as np
from sklearn.feature_extraction.text import CountVectorizer
from sklearn.metrics.pairwise import cosine_similarity

pd.set_option('display.float_format', '{:.4f}'.format)

df = pd.DataFrame({'text1': ['The quick brown fox jumps over the lazy dog',
                             'The red apple',
                             'The big blue sky'],
                   'text2': ['The lazy cat jumps over the brown dog',
                             'The red apple',
                             'The big yellow sun']})

vectorizer = CountVectorizer()

# np.hstack([df["text1"], df["text2"]]) puts all "text2" after "text1"
X = vectorizer.fit_transform(np.hstack([df["text1"], df["text2"]]))

cs = cosine_similarity(X)  # full symmetric numpy.ndarray

# The values you want are on an offset diagonal of cs since
# "text2" strings were stacked at the end of "text1" strings

pairwise_cs = cs.diagonal(offset=len(df))
df["cosine_similarity"] = pairwise_cs

print(df)

其显示：

text1                                  text2  cosine_similarity
0  The quick brown fox jumps over the lazy dog  The lazy cat jumps over the brown dog             0.8581
1                                The red apple                          The red apple             1.0000
2                             The big blue sky                     The big yellow sun             0.5000

赞(0）回复(0）举报 2023-03-06

我来回答

pandas 计算字符串之间的余弦相似度时未获得预期结果

1条答案

相关问题

热门标签

最新问答