Python:比较两个csv文件并打印出差异

1cosmwyk  于 2022-12-15  发布在  Python
关注(0)|答案(8)|浏览(178)

我需要比较两个CSV文件并打印出第三个CSV文件中的差异。在我的例子中,第一个CSV是名为old.csv的旧哈希列表,第二个CSV是包含旧哈希和新哈希的新哈希列表。
下面是我的代码:

import csv
t1 = open('old.csv', 'r')
t2 = open('new.csv', 'r')
fileone = t1.readlines()
filetwo = t2.readlines()
t1.close()
t2.close()

outFile = open('update.csv', 'w')
x = 0
for i in fileone:
    if i != filetwo[x]:
        outFile.write(filetwo[x])
    x += 1
outFile.close()

第三个文件是旧文件的副本,不是更新的。有什么问题吗?希望你能帮助我,非常感谢!!
PS:我不想使用diff

mgdq6dx1

mgdq6dx11#

问题是你要把fileone中的每一行与filetwo中的同一行进行比较。一旦一个文件中多出一行,你就会发现这两行永远不会相等。试试这个:

with open('old.csv', 'r') as t1, open('new.csv', 'r') as t2:
    fileone = t1.readlines()
    filetwo = t2.readlines()

with open('update.csv', 'w') as outFile:
    for line in filetwo:
        if line not in fileone:
            outFile.write(line)
gcmastyq

gcmastyq2#

您可能会发现此软件包很有用(csv-diff):

pip install csv-diff

安装后,您可以从命令行运行它:

csv-diff one.csv two.csv --key=id
yhqotfr8

yhqotfr83#

使用集合来检测差异感觉很自然。

#!/usr/bin/env python3

import sys
import argparse
import csv

def get_dataset(f):
    return set(map(tuple, csv.reader(f)))

def main(f1, f2, outfile, sorting_column):
    set1 = get_dataset(f1)
    set2 = get_dataset(f2)
    different = set1 ^ set2

    output = csv.writer(outfile)

    for row in sorted(different, key=lambda x: x[sorting_column], reverse=True):
        output.writerow(row)

if __name__ == '__main__':
    parser = argparse.ArgumentParser()

    parser.add_argument('infile', nargs=2, type=argparse.FileType('r'))
    parser.add_argument('outfile', nargs='?', type=argparse.FileType('w'), default=sys.stdout)
    parser.add_argument('-sc', '--sorting-column', nargs='?', type=int, default=0)

    args = parser.parse_args()

    main(*args.infile, args.outfile, args.sorting_column)
wztqucjr

wztqucjr4#

我假设你的新文件和你的旧文件一样,除了在旧文件之间添加了一些行,两个文件中的旧行以相同的顺序存储。
试试这个:

with open('old.csv', 'r') as t1:
    old_csv = t1.readlines()
with open('new.csv', 'r') as t2:
    new_csv = t2.readlines()

with open('update.csv', 'w') as out_file:
    line_in_new = 0
    line_in_old = 0
    while line_in_new < len(new_csv) and line_in_old < len(old_csv):
        if old_csv[line_in_old] != new_csv[line_in_new]:
            out_file.write(new_csv[line_in_new])
        else:
            line_in_old += 1
        line_in_new += 1

字符串

  • 注意,我使用了上下文管理器with和一些有意义的变量名,这使它更容易理解,并且您不需要csv包,因为您在这里不使用它的任何功能。
  • 关于你的代码,你几乎做了正确的事情,除了--你不能转到旧CSV的下一行,除非你在两个CSV中阅读了相同的内容。也就是说,如果你发现了一个新行,继续读新文件,直到你偶然发现了一个旧行,然后你就可以继续读了。
    **更新:**这个解决方案不如Chris Mueller's one那么漂亮,Chris Mueller's one对于小文件来说是完美的,非常Python化,但是它只读取文件一次(保持原来算法的想法),因此如果你有更大的文件,它会更好。
oug3syen

oug3syen5#

with open('first_test_pipe.csv', 'r') as t1, open('validation.csv', 'r') as t2:
    filecoming = t1.readlines()
    filevalidation = t2.readlines()

for i in range(0,len(filevalidation)):
    coming_set = set(filecoming[i].replace("\n","").split(","))
    validation_set = set(filevalidation[i].replace("\n","").split(","))
    ReceivedDataList=list(validation_set.intersection(coming_set))
    NotReceivedDataList=list(coming_set.union(validation_set)- 
    coming_set.intersection(validation_set))
    print(NotReceivedDataList)
byqmnocz

byqmnocz6#

import pandas as pd
import sys
import csv

def dataframe_difference(df1: pd.DataFrame, df2: pd.DataFrame, csvfile, which=None):
    """Find rows which are different between two DataFrames."""
    comparison_df = df1.merge(
        df2,
        indicator=True,
        how='outer'
    )
    if which is None:
        diff_df = comparison_df[comparison_df['_merge'] != 'both']
    else:
        diff_df = comparison_df[comparison_df['_merge'] == which]
    diff_df.to_csv(csvfile)
    return diff_df

if __name__ == '__main__':
    df1 = pd.read_csv(sys.argv[1], sep=',')    
    df2 = pd.read_csv(sys.argv[2], sep=',')

    df1.sort_values(sys.argv[3])
    df2.sort_values(sys.argv[3])
    #df1.drop(df1.columns[list(map(int, sys.argv[4].split()))], axis = 1, inplace = True)
    #df2.drop(df2.columns[list(map(int, sys.argv[4].split()))], axis = 1, inplace = True)

    print(dataframe_difference(df1, df2, sys.argv[5]))

使用运行

python3 script.py file1.csv file2.csv some_common_header_to_sort_each_file output_file.csv

如果要从比较中删除任何列,请取消注解df.drop part并运行

python3 script.py file1.csv file2.csv some_common_header_to_sort_each_file "x y z..." output_file.csv

其中x,y,z是要删除的列号,index从0开始。

wfveoks0

wfveoks07#

感谢@vishnoo-rath在上述答案之一下的评论,提供了以下页面的链接:https://github.com/simonw/csv-diff#as-a-python-library

from csv_diff import load_csv, compare
diff = compare(
    load_csv(open("one.csv"), key="id"),
    load_csv(open("two.csv"), key="id")
)
print(diff)
50pmv0ei

50pmv0ei8#

打开(“文件一. csv”,“r”)作为r1,打开(“文件二. csv”,“r”)作为r2,打开(“文件三. csv”,“w”,换行符=“”)作为r3:旧csv = r1.读取行()新csv = r2.读取行()

#For example , if you want to add headers in the thrid file , use like below.
fieldnames = ['Configuration','Host','IPAddress','Account','Profile','Version','Type','OS']
compare_csv = csv.DictWriter(r3, fieldnames=fieldnames)
compare_csv.writeheader()
for row2 in old_csv:
    if row2 not in new_csv:
        r3.write(row2)

相关问题