pandas 两个 Dataframe 的差异

3pmvbmvn  于 2023-03-28  发布在  其他
关注(0)|答案(8)|浏览(217)

我需要比较两个不同大小的 Dataframe ,并打印出不匹配的行。让我们采取以下两个:

df1 = DataFrame({
'Buyer': ['Carl', 'Carl', 'Carl'],
'Quantity': [18, 3, 5, ]})

df2 = DataFrame({
'Buyer': ['Carl', 'Mark', 'Carl', 'Carl'],
'Quantity': [2, 1, 18, 5]})

什么是最有效的方式来逐行遍历df2并打印出不在df1中的行,例如:

Buyer     Quantity 
Carl         2
Mark         1

重要提示:我不希望有行:

Buyer     Quantity 
Carl         3

包括在差异中:
我已经试过了:Comparing two dataframes of different length row by row and adding columns for each row with equal valueCompare two DataFrames and output their differences side-by-side
但这些与我的问题不匹配。

3zwjbxry

3zwjbxry1#

merge使用方法'outer'并传递参数indicator=True这将告诉您行是否存在于两个/left only/right only中,然后您可以过滤合并的df:

In [22]:
merged = df1.merge(df2, indicator=True, how='outer')
merged[merged['_merge'] == 'right_only']

Out[22]:
  Buyer  Quantity      _merge
3  Carl         2  right_only
4  Mark         1  right_only
6yoyoihd

6yoyoihd2#

你可能会发现这是最好的:

df2[ ~df2.isin(df1)].dropna()
23c0lvtd

23c0lvtd3#

@EdChum的答案是不言自明的。但是使用not 'both'条件更有意义,而且你不需要关心比较的顺序,这就是真实的的diff应该是什么样子的。为了回答你的问题:

merged = df1.merge(df2, indicator=True, how='outer')
merged.loc = [merged['_merge'] != 'both']
nszi6y05

nszi6y054#

diff = set(zip(df2.Buyer, df2.Quantity)) - set(zip(df1.Buyer, df1.Quantity))

这是我想到的第一个解决方案,然后你可以将diff集放回DF中进行展示。

g2ieeal7

g2ieeal76#

如果您只关心将新买家添加到其他df,请尝试以下操作:

df_delta=df2[df2['Buyer'].apply(lambda x: x not in df1['Buyer'].values)]
xcitsw88

xcitsw887#

一个重要的边缘用例

考虑下面的情况,在第二个 Dataframe 中有一个额外的重复条目。('Carl', 5)

df1 = DataFrame({ 'Buyer':    ['Carl', 'Carl', 'Carl'],
                  'Quantity': [   18 ,     3 ,     5 ]  })

df2 = DataFrame({ 'Buyer':    ['Carl', 'Mark', 'Carl', 'Carl', 'Carl'],
                  'Quantity': [    2 ,     1 ,    18 ,     5 ,     5 ]  })

EdChum的回答会给予你以下几点:

merged = df1.merge(df2, indicator=True, how='outer')
print(merged[merged['_merge'] == 'right_only'])

  Buyer  Quantity      _merge
4  Carl         2  right_only
5  Mark         1  right_only

正如您所看到的,解决方案忽略了额外的重复值,这取决于您正在做的事情是您想要避免的。
以下是一个更有可能实现您所需的解决方案:

df1['duplicate_counter'] = df1.groupby(list(df1.columns)).cumcount()
df2['duplicate_counter'] = df2.groupby(list(df2.columns)).cumcount()
merged = df1.merge(df2, indicator=True, how='outer')
merged[merged['_merge'] == 'right_only']

  Buyer  Quantity  duplicate_counter      _merge
3  Carl         2                  0  right_only
4  Mark         1                  0  right_only
5  Carl         5                  1  right_only

重复计数器确保每一行都是唯一的,这意味着不会删除重复的值。合并后,您可以删除duplicate_counter。

q3aa0525

q3aa05258#

还有datacompy。它允许导出一些基于字符串的比较报告,如下所示:

DataComPy Comparison
--------------------

DataFrame Summary
-----------------

  DataFrame  Columns  Rows
0  original        5     6
1       new        4     5

Column Summary
--------------

Number of columns in common: 4
Number of columns in original but not in new: 1
Number of columns in new but not in original: 0

Row Summary
-----------

Matched on: acct_id
Any duplicates on match values: Yes
Absolute Tolerance: 0.0001
Relative Tolerance: 0
Number of rows in common: 5
Number of rows in original but not in new: 1
Number of rows in new but not in original: 0

Number of rows with some compared columns unequal: 5
Number of rows with all compared columns equal: 0

Column Comparison
-----------------

Number of columns compared with some values unequal: 3
Number of columns compared with all values equal: 1
Total number of values which compare unequal: 7

Columns with Unequal Values or Types
------------------------------------

       Column original dtype new dtype  # Unequal  Max Diff  # Null Diff
0  dollar_amt        float64   float64          1    0.0500            0
1   float_fld        float64   float64          4    0.0005            3
2        name         object    object          2    0.0000            0

Sample Rows with Unequal Values
-------------------------------

       acct_id  dollar_amt (original)  dollar_amt (new)
0  10000001234                 123.45             123.4

       acct_id  float_fld (original)  float_fld (new)
0  10000001234            14530.1555        14530.155
5  10000001238                   NaN          111.000
2  10000001236                   NaN            1.000
1  10000001235                1.0000              NaN

       acct_id name (original)            name (new)
0  10000001234  George Maharis  George Michael Bluth
3  10000001237      Bob Loblaw         Robert Loblaw

Sample Rows Only in original (First 10 Columns)
-----------------------------------------------

       acct_id  dollar_amt           name  float_fld    date_fld
4  10000001238        1.05  Lucille Bluth        NaN  2017-01-01

相关问题