动态生成json文件密钥并在S3上写入

3htmauhk  于 2023-04-08  发布在  其他
关注(0)|答案(1)|浏览(110)

我正在使用Python脚本生成JSON文件，但for循环后的问题是它只拾取最后更新的值。下面是代码。
1读取水印文件：

watermark_file = config_dict["watermark_file"] + "watermark.json"
current_date, flag = read_watermark_file(config_dict.get("out_bucket"), watermark_file)
contents = list_s3_files(opt={'Bucket': config_dict['inp_bucket'], 'Prefix': config_dict['inp_location']})
print("contents :", contents)
for n in range(len(contents)):
    watermark_json = {}
    loop = {}
    zipped_fileName = contents[n].split("/")[-1]
    therapeutic_area = re.match("(.*?)_(.*)", zipped_fileName)[1]
    indication = re.match("(.*?)_(.*?)_(.*)", zipped_fileName)[2]
    print("value of n:", n)
    loop['item_' + str(n)] = {"therapeutic_area": therapeutic_area,
                              "indication": indication,
                              "s3_path": config_dict["inp_location"] + therapeutic_area + "/" + indication + "/"}
    print("loop :", loop)
    watermark_json.update(loop)
    print("watermark_json :", watermark_json)
# update water mark file
print("watermark_file :", watermark_file)
watermark_json['date_dir'] = datetime.datetime.now().strftime("%Y/%m/%d/%H") + "/"
watermark_json['processed_flag'] = False
print("final watermark file ", watermark_json)
# refresh watermark file
write_to_s3(config_dict['out_bucket'], watermark_file, watermark_json, config_dict)

日志：

2020-08-23T23:00:43.055+05:30

Copy
contents : ['mdit/cord/data/inbox/Immunology_COVID-19_Data_202008061200_09.zip', 'mdit/cord/data/inbox/Immunology_SLE_Data_202008131800_01.zip', 'mdit/cord/data/inbox/Neurology_ALZ_Data_202008031800_01.zip']
contents : ['mdit/cord/data/inbox/Immunology_COVID-19_Data_202008061200_09.zip', 'mdit/cord/data/inbox/Immunology_SLE_Data_202008131800_01.zip', 'mdit/cord/data/inbox/Neurology_ALZ_Data_202008031800_01.zip']

2020-08-23T23:00:43.055+05:30

Copy
value of n: 0
value of n: 0

2020-08-23T23:00:43.055+05:30

Copy
loop : {'item_0': {'therapeutic_area': 'Immunology', 'indication': 'COVID-19', 's3_path': 'mdit/cord/data/inbox/Immunology/COVID-19/'}}
loop : {'item_0': {'therapeutic_area': 'Immunology', 'indication': 'COVID-19', 's3_path': 'mdit/cord/data/inbox/Immunology/COVID-19/'}}

2020-08-23T23:00:43.055+05:30

Copy
watermark_json : {'item_0': {'therapeutic_area': 'Immunology', 'indication': 'COVID-19', 's3_path': 'mdit/cord/data/inbox/Immunology/COVID-19/'}}
watermark_json : {'item_0': {'therapeutic_area': 'Immunology', 'indication': 'COVID-19', 's3_path': 'mdit/cord/data/inbox/Immunology/COVID-19/'}}

2020-08-23T23:00:43.055+05:30

Copy
value of n: 1
value of n: 1

2020-08-23T23:00:43.055+05:30

Copy
loop : {'item_1': {'therapeutic_area': 'Immunology', 'indication': 'SLE', 's3_path': 'mdit/cord/data/inbox/Immunology/SLE/'}}
loop : {'item_1': {'therapeutic_area': 'Immunology', 'indication': 'SLE', 's3_path': 'mdit/cord/data/inbox/Immunology/SLE/'}}

2020-08-23T23:00:43.055+05:30

Copy
watermark_json : {'item_1': {'therapeutic_area': 'Immunology', 'indication': 'SLE', 's3_path': 'mdit/cord/data/inbox/Immunology/SLE/'}}
watermark_json : {'item_1': {'therapeutic_area': 'Immunology', 'indication': 'SLE', 's3_path': 'mdit/cord/data/inbox/Immunology/SLE/'}}

2020-08-23T23:00:43.055+05:30

Copy
value of n: 2
value of n: 2

2020-08-23T23:00:43.055+05:30

Copy
loop : {'item_2': {'therapeutic_area': 'Neurology', 'indication': 'ALZ', 's3_path': 'mdit/cord/data/inbox/Neurology/ALZ/'}}
loop : {'item_2': {'therapeutic_area': 'Neurology', 'indication': 'ALZ', 's3_path': 'mdit/cord/data/inbox/Neurology/ALZ/'}}

2020-08-23T23:00:43.055+05:30

Copy
watermark_json : {'item_2': {'therapeutic_area': 'Neurology', 'indication': 'ALZ', 's3_path': 'mdit/cord/data/inbox/Neurology/ALZ/'}}
watermark_json : {'item_2': {'therapeutic_area': 'Neurology', 'indication': 'ALZ', 's3_path': 'mdit/cord/data/inbox/Neurology/ALZ/'}}

2020-08-23T23:00:43.055+05:30

Copy
watermark_file : mdit/cord/technical_metadata/watermark/watermark.json
watermark_file : mdit/cord/technical_metadata/watermark/watermark.json

2020-08-23T23:00:43.055+05:30

Copy
final watermark file 
 {'item_2': {'therapeutic_area': 'Neurology', 'indication': 'ALZ', 's3_path': 'mdit/cord/data/inbox/Neurology/ALZ/'}, 'date_dir': '2020/08/23/17/', 'processed_flag': False}

需要的Watermark.json文件：

{
    "loop": {
        "item_0":{
                "therapeutic_area": "Immunology",
                "indication": "SLE",
                "s3_path": "mdit/cord/data/inbound/Immunology/SLE/"
            },
        "item_1":{
                "therapeutic_area": "Immunology",
                "indication": "COVID-19",
                "s3_path": "mdit/cord/data/inbound/Immunology/COVID-19/"
            },
        "item_2":{
                "therapeutic_area": "Neurology",
                "indication": "ALZ",
                "s3_path": "mdit/cord/data/inbound/Immunology/ALZ/"
            }
    },
    "date_dir": "2020/08/23/12/",
    "processed_flag": false
}

Json文件从代码中获取生成：

{
    "item_2": {
        "therapeutic_area": "Neurology",
        "indication": "ALZ",
        "s3_path": "mdit/cord/data/inbox/Neurology/ALZ/"
    },
    "date_dir": "2020/08/23/17/",
    "processed_flag": false
}

我在代码中做错了什么？

JSON

1条答案

按热度按时间
c8ib6hqw

c8ib6hqw1#

导致代码错误的原因是watermark_json = {}位于for n in range(len(contents)):循环中。它应该位于for循环之前。
代码还需要进一步修改以获得所需的输出。
你可以试试下面的代码：

watermark_file = config_dict["watermark_file"] + "watermark.json"
current_date, flag = read_watermark_file(config_dict.get("out_bucket"), watermark_file)
contents = list_s3_files(opt={'Bucket': config_dict['inp_bucket'], 'Prefix': config_dict['inp_location']})
print("contents :", contents)
watermark_json = {'loop': {}}  # <- This line is changed
for n in range(len(contents)):
    loop = {}
    zipped_fileName = contents[n].split("/")[-1]
    therapeutic_area = re.match("(.*?)_(.*)", zipped_fileName)[1]
    indication = re.match("(.*?)_(.*?)_(.*)", zipped_fileName)[2]
    print("value of n:", n)
    loop['item_' + str(n)] = {"therapeutic_area": therapeutic_area,
                              "indication": indication,
                              "s3_path": config_dict["inp_location"] + therapeutic_area + "/" + indication + "/"}
    print("loop :", loop)
    watermark_json['loop'].update(loop)  # <- This line is changed
    print("watermark_json :", watermark_json)
# update water mark file
print("watermark_file :", watermark_file)
watermark_json['date_dir'] = datetime.datetime.now().strftime("%Y/%m/%d/%H") + "/"
watermark_json['processed_flag'] = False
print("final watermark file ", watermark_json)
# refresh watermark file
write_to_s3(config_dict['out_bucket'], watermark_file, watermark_json, config_dict)
赞(0）分享  回复(0）举报  2023-04-08
我来回答

相关问题

    查看更多

    热门标签

    更多
    JavaquerypythonNode开发语言requestUtil数据库Table后端算法LoggerMessageElementParser

    最新问答

    更多
    • 蜀ICP备2022004421号-2 NULL123 https://www.null123.com © All rights reserved
    • 本站内容来源互联网,如果侵犯您的权益请联系我们删除, 联系方式：448109455@qq.com