pandas Python添加与列值关联的权重

zlwx9yxi  于 2023-01-28  发布在  Python
关注(0)|答案(2)|浏览(133)

我正在处理一个超大的数据库。下面是一个示例:

import pandas as pd
import numpy as np
df = pd.DataFrame({ 
'ID': ['A', 'A', 'A', 'X', 'X', 'Y'], 
})
 ID
0  A
1  A
2  A
3  X
4  X
5  Y

现在,给定列“”“ID”“”中每个值的频率,我想使用下面的函数计算权重,并添加一列,该列的权重与“”“ID”"“中的每个值相关联。

def get_weights_inverse_num_of_samples(label_counts, power=1.):
    no_of_classes = len(label_counts)
    weights_for_samples = 1.0/np.power(np.array(label_counts), power)
    weights_for_samples = weights_for_samples/ np.sum(weights_for_samples)*no_of_classes
    return weights_for_samples

freq = df.value_counts()
print(freq)
ID
A     3
X     2
Y     1

weights = get_weights_inverse_num_of_samples(freq)
print(weights)
[0.54545455 0.81818182 1.63636364]

因此,我正在寻找一种有效的方法来获得这样的 Dataframe 给定上述权重:

ID  sample_weight
0  A   0.54545455
1  A   0.54545455
2  A   0.54545455
3  X   0.81818182
4  X   0.81818182
5  Y   1.63636364
8yparm6h

8yparm6h1#

如果您更多地依赖duck-typing,则可以重写函数以返回与输出相同的输入类型。
这将使您不必在调用.map之前显式地返回到.index

import pandas as pd

df = pd.DataFrame({'ID': ['A', 'A', 'A', 'X', 'X', 'Y'})

def get_weights_inverse_num_of_samples(label_counts, power=1):
    """Using object methods here instead of coercing to numpy ndarray"""

    no_of_classes = len(label_counts)
    weights_for_samples = 1 / (label_counts ** power)
    return weights_for_samples / weights_for_samples.sum() * no_of_classes

# select the column before using `.value_counts()`
#   this saves us from ending up with a `MultiIndex` Series
freq = df['ID'].value_counts() 

weights = get_weights_inverse_num_of_samples(freq)

print(weights)
# A    0.545455
# X    0.818182
# Y    1.636364

# note that now our weights are still a `pd.Series` 
#  that we can align directly against our `"ID"` column

df['sample_weight'] = df['ID'].map(weights)

print(df)
#   ID  sample_weight
# 0  A       0.545455
# 1  A       0.545455
# 2  A       0.545455
# 3  X       0.818182
# 4  X       0.818182
# 5  Y       1.636364
6za6bjd0

6za6bjd02#

您可以map这些值:

df['sample_weight'] = df['ID'].map(dict(zip(freq.index.get_level_values(0), weights)))
  • 注意:value_counts返回一个单级别的MultiIndex,因此需要get_level_values。*

如@ScottBoston所述,更好的方法是使用:

freq = df['ID'].value_counts()

df['sample_weight'] = df['ID'].map(dict(zip(freq.index, weights)))

输出:

ID  sample_weight
0  A       0.545455
1  A       0.545455
2  A       0.545455
3  X       0.818182
4  X       0.818182
5  Y       1.636364

相关问题