panda Dataframe 中的groupby加权平均与求和

lxkprmvk  于 2022-12-24  发布在  其他
关注(0)|答案(8)|浏览(267)

我有一个 Dataframe :

Out[78]: 
   contract month year  buys  adjusted_lots    price
0         W     Z    5  Sell             -5   554.85
1         C     Z    5  Sell             -3   424.50
2         C     Z    5  Sell             -2   424.00
3         C     Z    5  Sell             -2   423.75
4         C     Z    5  Sell             -3   423.50
5         C     Z    5  Sell             -2   425.50
6         C     Z    5  Sell             -3   425.25
7         C     Z    5  Sell             -2   426.00
8         C     Z    5  Sell             -2   426.75
9        CC     U    5   Buy              5  3328.00
10       SB     V    5   Buy              5    11.65
11       SB     V    5   Buy              5    11.64
12       SB     V    5   Buy              2    11.60

我需要一个总和adjusted_lots,价格是加权平均值,价格和adjusted_lots,由所有其他列分组,即分组(合同,月,年和购买)
R上类似的解决方案可以通过下面的代码实现,使用dplyr,但是不能在Pandas上做同样的事情。

> newdf = df %>%
  select ( contract , month , year , buys , adjusted_lots , price ) %>%
  group_by( contract , month , year ,  buys) %>%
  summarise(qty = sum( adjusted_lots) , avgpx = weighted.mean(x = price , w = adjusted_lots) , comdty = "Comdty" )

> newdf
Source: local data frame [4 x 6]

  contract month year comdty qty     avgpx
1        C     Z    5 Comdty -19  424.8289
2       CC     U    5 Comdty   5 3328.0000
3       SB     V    5 Comdty  12   11.6375
4        W     Z    5 Comdty  -5  554.8500

通过groupby或任何其他解决方案是否也可以实现相同的效果?

a6b3iqyw

a6b3iqyw1#

**编辑:**更新聚合,使其适用于最新版本的panda

要将多个函数传递给groupby对象,需要传递一个元组以及聚合函数和该函数所应用的列:

# Define a lambda function to compute the weighted mean:
wm = lambda x: np.average(x, weights=df.loc[x.index, "adjusted_lots"])

# Define a dictionary with the functions to apply for a given column:
# the following is deprecated since pandas 0.20:
# f = {'adjusted_lots': ['sum'], 'price': {'weighted_mean' : wm} }
# df.groupby(["contract", "month", "year", "buys"]).agg(f)

# Groupby and aggregate with namedAgg [1]:
df.groupby(["contract", "month", "year", "buys"]).agg(adjusted_lots=("adjusted_lots", "sum"),  
                                                      price_weighted_mean=("price", wm))

                          adjusted_lots  price_weighted_mean
contract month year buys                                    
C        Z     5    Sell            -19           424.828947
CC       U     5    Buy               5          3328.000000
SB       V     5    Buy              12            11.637500
W        Z     5    Sell             -5           554.850000

您可以在这里看到更多:

  • http://pandas.pydata.org/pandas-docs/stable/groupby.html#applying-multiple-functions-at-once

在这个类似的问题中

[1]网址:https://pandas.pydata.org/pandas-docs/stable/whatsnew/v0.25.0.html#groupby-aggregation-with-relabeling

bksxznpy

bksxznpy2#

用groupby(...).apply(...)进行加权平均可能会非常慢(比下面慢100倍),请参见我的答案(以及其他答案)。

def weighted_average(df,data_col,weight_col,by_col):
    df['_data_times_weight'] = df[data_col]*df[weight_col]
    df['_weight_where_notnull'] = df[weight_col]*pd.notnull(df[data_col])
    g = df.groupby(by_col)
    result = g['_data_times_weight'].sum() / g['_weight_where_notnull'].sum()
    del df['_data_times_weight'], df['_weight_where_notnull']
    return result
5hcedyr0

5hcedyr03#

这样做不是简单得多吗。
1.将(adjusted_lots * price_weighted_mean)乘以新列“X”
1.使用groupby().sum()对列“X”和“adjusted_lots”进行分组df_grouped
1.将df_grouped的加权平均值计算为df_grouped ['X']/df_grouped ['adjusted_lots']

zdwk9cvp

zdwk9cvp4#

使用dict聚合函数的解决方案将在panda的未来版本(0.22版)中弃用:

FutureWarning: using a dict with renaming is deprecated and will be removed in a future 
version return super(DataFrameGroupBy, self).aggregate(arg, *args, **kwargs)

使用groupby apply并返回Series来重命名列,如中所述:Rename result columns from Pandas aggregation ("FutureWarning: using a dict with renaming is deprecated")

def my_agg(x):
    names = {'weighted_ave_price': (x['adjusted_lots'] * x['price']).sum()/x['adjusted_lots'].sum()}
    return pd.Series(names, index=['weighted_ave_price'])

会产生同样的结果:

>df.groupby(["contract", "month", "year", "buys"]).apply(my_agg)

                          weighted_ave_price
contract month year buys                    
C        Z     5    Sell          424.828947
CC       U     5    Buy          3328.000000
SB       V     5    Buy            11.637500
W        Z     5    Sell          554.850000
irlmq6kh

irlmq6kh5#

使用datar,您不必学习panda API来转换R代码:

>>> from datar.all import f, tibble, c, rep, select, summarise, sum, weighted_mean, group_by
>>> df = tibble(
...     contract=c('W', rep('C', 8), 'CC', rep('SB', 3)),
...     month=c(rep('Z', 9), 'U', rep('V', 3)),
...     year=5,
...     buys=c(rep('Sell', 9), rep('Buy', 4)),
...     adjusted_lots=[-5, -3, -2, -2, -3, -2, -3, -2, -2, 5, 5, 5, 2],
...     price=[554.85, 424.50, 424.00, 423.75, 423.50, 425.50, 425.25, 426.00, 426.75,3328.00, 11.65, 11.64, 1
1.60]
... )
>>> df
   contract month  year  buys  adjusted_lots    price
0         W     Z     5  Sell             -5   554.85
1         C     Z     5  Sell             -3   424.50
2         C     Z     5  Sell             -2   424.00
3         C     Z     5  Sell             -2   423.75
4         C     Z     5  Sell             -3   423.50
5         C     Z     5  Sell             -2   425.50
6         C     Z     5  Sell             -3   425.25
7         C     Z     5  Sell             -2   426.00
8         C     Z     5  Sell             -2   426.75
9        CC     U     5   Buy              5  3328.00
10       SB     V     5   Buy              5    11.65
11       SB     V     5   Buy              5    11.64
12       SB     V     5   Buy              2    11.60
>>> newdf = df >> \
...   select(f.contract, f.month, f.year, f.buys, f.adjusted_lots, f.price) >> \
...   group_by(f.contract, f.month, f.year, f.buys) >> \
...   summarise(
...       qty = sum(f.adjusted_lots), 
...       avgpx = weighted_mean(x = f.price , w = f.adjusted_lots), 
...       comdty = "Comdty"
...   )
[2021-05-24 13:11:03][datar][   INFO] `summarise()` has grouped output by ['contract', 'month', 'year'] (overr
ide with `_groups` argument)
>>> 
>>> newdf
  contract month  year  buys  qty        avgpx  comdty
0        C     Z     5  Sell  -19   424.828947  Comdty
1       CC     U     5   Buy    5  3328.000000  Comdty
2       SB     V     5   Buy   12    11.637500  Comdty
3        W     Z     5  Sell   -5   554.850000  Comdty
[Groups: ['contract', 'month', 'year'] (n=4)]

我是软件包的作者。如果您有任何问题,请随时提交问题。

gab6jxml

gab6jxml6#

ErnestScribbler的答案比公认的答案要快得多。下面是一个多元类比:

def weighted_average(df,data_col,weight_col,by_col):
    ''' Now data_col can be a list of variables '''
    df_data = df[data_col].multiply(df[weight_col], axis='index')
    df_weight = pd.notnull(df[data_col]).multiply(df[weight_col], axis='index')
    df_data[by_col] = df[by_col]
    df_weight[by_col] = df[by_col]    
    result = df_data.groupby(by_col).sum() / df_weight.groupby(by_col).sum()
    return result
oxalkeyp

oxalkeyp7#

当我遇到类似的问题时,我遇到了这个线程。在我的情况下,我想生成一个四分卫评级的加权指标,如果不止一个四分卫在给定的NFL比赛中尝试传球。
如果在扩展时遇到重大的性能问题,我可能会更改代码。目前,我更喜欢将解决方案与其他转换一起压缩到.agg函数中。很高兴看到有人有更简单的解决方案来实现相同的目的。最终,我采用了闭包模式。
闭包方法的神奇之处在于,如果这对未来的读者来说是一种不熟悉的模式,我仍然可以向panda的.agg()方法返回一个简单的函数,但是我可以使用从顶层factory函数预先配置的一些附加信息来完成。

def weighted_mean_factory(*args, **kwargs):
    weights = kwargs.get('w').copy()
    
    def weighted_mean(x):
        x_mask = ~np.isnan(x)
        w = weights.loc[x.index]
        
        if all(v is False for v in x_mask):
            raise ValueError('there are no non-missing x variable values')

        return np.average(x[x_mask], weights=w[x_mask])
    
    return weighted_mean

res_df = df.groupby(['game_id', 'team'])\
    .agg(pass_player_cnt=('attempts', count_is_not_zero),
         completions=('completions', 'sum'), 
         attempts=('attempts', 'sum'),
         pass_yds=('pass_yards', 'sum'),
         pass_tds=('pass_tds', 'sum'), 
         pass_int=('pass_int', 'sum'), 
         sack_taken=('sacks_taken', 'sum'), 
         sack_yds_loss=('sack_yds_loss', 'sum'), 
         longest_completion=('longest_completion', 'max'),
         qbr_w_avg=('qb_rating', weighted_mean_factory(x='qb_rating', w=df['attempts']))
         )

下面是形状为(5436,31)的DataFrame上的一些基本基准测试统计数据,在此阶段,就性能而言,这些统计数据在我看来并不重要:

149 ms ± 4.75 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
x4shl7ld

x4shl7ld8#

这将original approach by jrjcclosure approach by MB结合在一起,其优点是能够重用闭包函数。

import pandas as pd

def group_weighted_mean_factory(df: pd.DataFrame, weight_col_name: str):
    # Ref: https://stackoverflow.com/a/69787938/
    def group_weighted_mean(x):
        try:
            return np.average(x, weights=df.loc[x.index, weight_col_name])
        except ZeroDivisionError:
            return np.average(x)
    return group_weighted_mean

df = ...  # Define
group_weighted_mean = group_weighted_mean_factory(df, "adjusted_lots")
g = df.groupby(...)  # Define
agg_df = g.agg({'price': group_weighted_mean})

相关问题